Clozure CL Documentation


1. About Clozure CL
1.1. Introduction to Clozure CL
2. Obtaining, Installing, and Running Clozure CL
2.1. Releases and System Requirements
2.2. Obtaining Clozure CL
2.3. Command Line Set Up
2.4. Personal Customization with the Init File
2.5. Command Line Options
2.6. Using Clozure CL with GNU Emacs and SLIME
2.7. Example Programs
3. Building Clozure CL from its Source Code
3.1. Building Definitions
3.2. Setting Up to Build
3.3. Kernel Build Prerequisites
3.4. Building Everything
3.5. Building the Kernel
3.6. Building the Heap Image
4. Using Clozure CL
4.1. Introduction
4.2. Trace
4.3. Advising
4.4. Directory
4.5. Unicode
4.6. Pathnames
4.7. Memory-mapped Files
4.8. Static Variables
4.9. Saving Applications
4.10. Concatenating FASL Files
4.11. Floating Point Numbers
4.12. Watched Objects
4.13. Code Coverage
4.14. Other Extensions
5. The Clozure CL IDE
5.1. Introduction
5.2. Building the IDE
5.3. Running the IDE
5.4. IDE Features
5.5. IDE Sources
5.6. The Application Builder
6. Hemlock Programming
6.1. Introduction
6.2. Representation of Text
6.3. Buffers
6.4. Altering and Searching Text
6.5. The Current Environment
6.6. Hemlock Variables
6.7. Commands
6.8. Modes
6.9. Character Attributes
6.10. Controlling the Display
6.11. Logical Key-Events
6.12. The Echo Area
6.13. Files
6.14. Hemlock's Lisp Environment
6.15. High-Level Text Primitives
6.16. Utilities
6.17. Miscellaneous
7. Programming with Threads
7.1. Threads Overview
7.2. (Intentionally) Missing Functionality
7.3. Implementation Decisions and Open Questions
7.4. Porting Code from the Old Thread Model
7.5. Background Terminal Input
7.6. The Threads which Clozure CL Uses for Its Own Purposes
7.7. Threads Dictionary
8. Programming with Sockets
8.1. Overview
8.2. Sockets Dictionary
9. Running Other Programs as Subprocesses
9.1. Overview
9.2. Examples
9.3. Limitations and known bugs
9.4. External-Program Dictionary
10. Streams
10.1. Stream Extensions
10.2. Creating Your Own Stream Classes with Gray Streams
10.3. Lisp Standard Streams and OS Standard Streams
11. Writing Portable Extensions to the Object System using the MetaObject Protocol
11.1. Overview
11.2. Implementation status
11.3. Concurrency issues
12. Profiling
12.1. Using the Linux oprofile system-level profiler
12.2. Using Apple's CHUD metering tools
13. The Foreign-Function Interface
13.1. Specifying And Using Foreign Types
13.2. Foreign Function Calls
13.3. Referencing and Using Foreign Memory Addresses
13.4. The Interface Database
13.5. Using Interface Directories
13.6. Using Shared Libraries
13.7. The Interface Translator
13.8. Case-sensitivity of foreign names in CCL
13.9. Reading Foreign Names
13.10. Tutorial: Using Basic Calls and Types
13.11. Tutorial: Allocating Foreign Data on the Lisp Heap
13.12. The Foreign-Function-Interface Dictionary
14. The Objective-C Bridge
14.1. Changes in 1.2
14.2. Using Objective-C Classes
14.3. Instantiating Objective-C Objects
14.4. Calling Objective-C Methods
14.5. Defining Objective-C Classes
14.6. Defining Objective-C Methods
14.7. Loading Frameworks
14.8. How Objective-C Names are Mapped to Lisp Symbols
15. Platform-specific notes
15.1. Overview
15.2. Unix/Posix/Darwin Features
15.3. Cocoa Programming in Clozure CL
15.4. Building an Application Bundle
15.5. Recommended Reading
15.6. Operating-System Dictionary
16. Understanding and Configuring the Garbage Collector
16.1. Heap space allocation
16.2. The Ephemeral GC
16.3. GC Page reclamation policy
16.4. "Pure" areas are read-only, paged from image file
16.5. Weak References
16.6. Weak References Dictionary
16.7. Garbage-Collection Dictionary
17. Implementation Details of Clozure CL
17.1. Threads and exceptions
17.2. Register usage and tagging
17.3. Heap Allocation
17.4. GC details
17.5. The ephemeral GC
17.6. Fasl files
17.7. The Objective-C Bridge
18. Modifying Clozure CL
18.1. Contributing Code Back to the Clozure CL Project
18.2. Using Clozure CL in "development" and in "user" mode
18.3. The Kernel Debugger
18.4. Using AltiVec in Clozure CL LAP functions
18.5. Development-Mode Dictionary
19. Questions and Answers
19.1. How can I do nonblocking (aka "unbuffered" and "raw") IO?
19.2. I'm using the graphics demos. Why doesn't the menubar change?
19.3. I'm using Slime and Cocoa. Why doesn't *standard-output* seem to work?
Glossary of Terms
Symbol Index

List of Tables

3.1. Platform-specific filename conventions
4.1. Line Termination Keywords

Chapter 1. About Clozure CL

1.1. Introduction to Clozure CL

Clozure CL is a fast, mature, open source Common Lisp implementation that runs on Linux, Mac OS X, FreeBSD, and Windows. Clozure CL was forked from Macintosh Common Lisp (MCL) in 1998 and the development has been entirely separate since.

When it was forked from MCL in 1998, the new Lisp was named OpenMCL. Subsequently, Clozure renamed its Lisp to Clozure CL, partly because its ancestor MCL has been released as open source. Clozure thought it might be confusing for users if there were two independent open-source projects with such similar names. The new name also reflects Clozure CL's current status as the flagship product of Clozure Associates.

Furthermore, the new name refers to Clozure CL's ancestry: in its early years, MCL was known as Coral Common Lisp, or "CCL". For years the package that contains most of Clozure CL's implementation-specific symbols has been named "CCL", an acronym that once stood for the name of the Lisp product. It seems fitting that "CCL" once again stands for the name of the product.

Some commands and source files may still refer to "OpenMCL" instead of Clozure CL.

Clozure CL compiles to native code and supports multithreading using native OS threads. It includes a foreign-function interface, and supports both Lisp code that calls external code, and external code that calls Lisp code. Clozure CL can create standalone executables on all supported platforms.

On Mac OS X, Clozure CL supports building GUI applications that use OS X's native Cocoa frameworks, and the OS X distributions include an IDE written with Cocoa, and distributed with complete sources.

On all supported platforms, Clozure CL can run as a command-line process, or as an inferior Emacs process using either SLIME or ILISP.

Features of Clozure CL include

  • Very fast compilation speed.

  • A fast, precise, compacting, generational garbage collector written in hand-optimized C. The sizes of the generations are fully configurable. Typically, a generation can be collected in a millisecond on modern systems.

  • Fast execution speed, competitive with other Common Lisp implementations on most benchmarks.

  • Robust and stable. Customers report that their CPU-intensive, multi-threaded applications run for extended periods on Clozure CL without difficulty.

  • Full native OS threads on all platforms. Threads are automatically distributed across multiple cores. The API includes support for shared memory, locking, and blocking for OS operations such as I/O.

  • Full Unicode support.

  • Full SLIME integration.

  • An IDE on Mac OS X, fully integrated with the Macintosh window system and User Interface standards.

  • Excellent debugging facilities. The names of all local variables are available in a backtrace.

  • A complete, mature foreign function interface, including a powerful bridge to Objective-C and Cocoa on Mac OS X.

  • Many extensions including: files mapped to Common Lisp vectors for fast file I/O; thread-local hash tables and streams to eliminate locking overhead; cons hashing support; and much more

  • Very efficient use of memory

Although it's an open-source project, available free of charge under a liberal license, Clozure CL is also a fully-supported product of Clozure Associates. Clozure continues to extend, improve, and develop Clozure CL in response to customer and user needs, and offers full support and development services for Clozure CL.

Chapter 2. Obtaining, Installing, and Running Clozure CL

2.1. Releases and System Requirements

As of this writing, Clozure CL 1.7 is the latest release; it was made in August 2011. For up-to-date information about releases, please see http://ccl.clozure.com/.

Clozure CL 1.7 runs on the following platforms:

  • Linux (x86, x86-64, ppc32, ppc64, armv7)

  • Mac OS X 10.5 and later (x86, x86-64)

  • FreeBSD 6.x and later (x86, x86-64)

  • Solaris (x86, x86-64)

  • Microsoft Windows XP and later (x86, x86-64)

2.1.1. 32- vs 64-bit versions

Naturally, 64-bit versions of Clozure CL require 64-bit processors, for example, a G5 or Core 2. Some early Intel-based Macintoshes used processors that don't support 64-bit operation, so the 64-bit Clozure CL will not run on them, although the 32-bit Clozure CL will.

2.1.2. 32-bit x86 versions require SSE2

The 32-bit x86 versions of Clozure CL depend on the presence of the SSE2 instructions. Most x86 processors manufactured and sold in the last several years support SSE2 (all Apple Intel-based Macs do, for instance), but there are some exceptions. The Wikipedia article on SSE2 lists processor models that support SSE2 (and also mentions some of the more notable exceptions).

2.1.3. LinuxPPC

Clozure CL requires version 2.2.13 (or later) of the Linux kernel and version 2.1.3 (or later) of the GNU C library (glibc) at a bare minimum.

2.1.4. Linux x86

Because of the nature of Linux distributions, it's difficult to give precise version number requirements. In general, a "fairly modern" (no more than 2 or three years old) kernel and C library are more likely to work well than older versions.

2.1.5. Linux ARM v7

The Linux ARM port is relatively new and is still a work-in-progress. Clozure CL needs some features (such as hardware floating-point, locking and memory-serialization primitives) that are only found in chips that implement architecture version 7 (ARMv7); technically, it needs the ARMv7 "application profile", which is sometimes called ARMv7a. In practice, most ARM consumer devices released in the last few years implement ARMv7, but there are exceptions, and it is not practical to enumerate all of the ARM devices that CCL should run on.

In addition to hardware issues, Clozure CL expects Linux to run in little-endian mode and expects software to follow "soft float" calling conventions. The latter has to do with how C functions accept floating-point arguments and return floating-point values.

2.1.6. FreeBSD x86

Clozure CL should run on FreeBSD 6.x and 7.x. FreeBSD 7 users will need to install the "compat6x" package in order to use the distributed Clozure CL kernel, which is built on a FreeBSD 6.x system.

2.1.7. Mac OS X x86

Clozure CL 1.7 runs on Mac OS X (x86) versions 10.5 and later, including 10.7 (Lion),

Clozure CL 1.6 runs on Mac OS X PPC as well as x86 processors.

2.1.8. Microsoft Windows

For Clozure CL versions prior to 1.7, the 32-bit executable does not run under 64-bit Windows.

2.2. Obtaining Clozure CL

There are three ways to obtain Clozure CL. For Mac OS X, there are disk images that can be used to install Clozure CL in the usual Macintosh way. For other OSes, Subversion is the best way to obtain Clozure CL. Mac OS X users can also use Subversion if they prefer. Tarballs are available for those who prefer them, but if you have Subversion installed, it is simpler and more flexible to use Subversion than tarballs.

There are three popular ways to use Clozure CL: as a stand-alone double-clickable application (Mac OS X only), as a command-line application, or with Emacs and SLIME.

The following sections describe these options.

2.2.1. The Mac Way

If you are using Mac OS X then you can install and use Clozure CL in the usual Macintosh way. Download and mount a disk image, then drag the ccl folder to the Applications folder or wherever you wish. After that you can double-click the Clozure CL application found inside the ccl directory. The disk images for version 1.7 are available at ftp://clozure.com/pub/release/1.7/

So that Clozure CL can locate its source code, and for other reasons explained in Section 4.6.2, “Predefined Logical Hosts”, you keep the Clozure CL application in the ccl directory. If you use a shell, you can set the value of the CCL_DEFAULT_DIRECTORY environment variable to explicitly indicate the location of the ccl directory. If you choose to do that, then the ccl directory and the Clozure CL application can each be in any location you find convenient.

2.2.2. Tarballs

Tarball distributions of Clozure CL release version 1.7 are available at ftp://clozure.com/pub/release/1.7/. Download and extract one on your local disk. Then edit the Clozure CL shell script to set the value of CCL_DEFAULT_DIRECTORY and start up the appropriate Clozure CL kernel. See Section 2.3.1, “The ccl Shell Script” for more information about the Clozure CL shell scripts.

2.2.3. Getting Clozure CL with Subversion

It is very easy to download and configure Clozure CL to obtain sources from the Subversion repository. This is the preferred way to get either the latest, or a specific version of Clozure CL, unless you prefer the Mac Way. Subversion is a source code control system that is in wide use. Many OSes come with Subversion pre-installed. A complete, buildable and runnable set of Clozure CL sources and binaries can be retrieved with a single Subversion command.

Note

Unless stated otherwise, examples in this chapter are given for Mac OS X in particular or Unix-based host environments in general.

For Windows, special care must be taken to install a working development environment. For more information see the Clozure CL Wiki at URL: http://trac.clozure.com/ccl/wiki/WindowsNotes

2.2.3.1. Checking Subversion Installation

Make sure that Subversion is installed on your system. Bring up a command line shell and type:

shell> svn
        

If Subversion is installed, you will see something like:

Type 'svn help' for usage
        

If Subversion is not installed, you will see something like:

-bash: svn: command not found
        

If Subversion is not installed, you'll need to figure out how to install it on your OS. You can find information about obtaining and installing Subversion at the Subversion web page.

2.2.3.2. Downloading Clozure CL Using Subversion

Before you download Clozure CL you should consider: Do you want to run the most recent source code, or the current stable release version? If you don't know how to answer this question, then you probably want the release version.

2.2.3.2.1. Downloading the Trunk

Day-to-day development of Clozure CL takes place in an area of the Subversion repository known as "the trunk". At most times, the trunk is perfectly usable, but occasionally it can be unstable or totally broken. If you wish to live on the bleeding edge, download sources from the trunk.

For example, the following command will fetch a copy of the trunk for Mac OS X (Darwin) with x86 processors (both 32- and 64-bit versions):

svn co http://svn.clozure.com/publicsvn/openmcl/trunk/darwinx86/ccl
         

To get a trunk Clozure CL for another platform, replace "darwinx86" with one of the following names (all versions include both 32- and 64-bit binaries):

  • darwinx86

  • linuxx86

  • freebsdx86

  • solarisx86

  • windows

  • linuxppc

  • darwinppc

2.2.3.2.2. Downloading a Release Version

Release versions of Clozure CL are intended to be stable. While bugs will be fixed in the release branches, enhancements and new features will go into the trunk. If you wish to run the stable release, the following command will fetch a copy of the release version 1.7 for Mac OS X (Darwin) with x86 processors (both 32- and 64-bit versions):

svn co http://svn.clozure.com/publicsvn/openmcl/release/1.7/darwinx86/ccl
          

To get the release version of Clozure CL for another platform, replace "darwinx86" with one of the following names:

  • darwinx86

  • linuxx86

  • freebsdx86

  • solarisx86

  • windows

  • linuxppc

  • darwinppc

These distributions contain complete sources and binaries. They use Subversion's "externals" features to share common sources; the majority of source code is the same across all versions.

2.2.4. Rebuilding Clozure CL From Sources

This section explains how to peform a "full rebuild" of Clozure CL from a source distribution.

After downloading Clozure CL sources, you should rebuild Clozure CL as described here. At the start of a full rebuild, object files in the ccl directory are deleted, which causes the build script to recompile the runtime kernel (C code) and high-level sources (Lisp), then save a new heap image. Doing a full rebuild helps to ensure that your local installation will run properly for your host OS environment.

In an interactive shell, a command sequence like the following will rebuild Clozure CL in place:

joe> cd /path/to/installed/ccl
joe:ccl> ./kernel-filename --no-init
Welcome to Clozure Common Lisp Version [...]
? (rebuild-ccl :full t)
<...lots of compilation output...>
? (quit)
joe:ccl>
    

Replace /path/to/installed/ccl with the path of the ccl directory that you downloaded.

Replace kernel-filename with the filename of the Lisp kernel program. To find the filename of a Lisp kernel image for your particular platform, see Section 3.1.1, “Platform-specific filename conventions”.

Specifying the --no-init option ensures that personal initializations do not interfere with rebuilding Clozure CL.

The rest of this section covers the following topics in brief:

Note

This section does not provide comprehensive documentation on the build process. Please refer to Chapter 3, Building Clozure CL from its Source Code for more information. Those more detailed instructions are used mainly by developers who maintain, customize, and/or port Clozure CL. If you are customizing Clozure CL, or if you run into some exceptional situation, you may need to perform the individual build steps.

2.2.4.1. Software Requirements for Building Clozure CL

In order to build Clozure CL you must have a working system and development environment. There are different requirements and setup procedures for each platform, but the main requirement is to have a C compiler and a few other utilities: GNU gcc or cc with ld and as; make; and m4. Please refer to Chapter 3, Building Clozure CL from its Source Code for details.

Note

If you don't have the prerequisite C compiler toolchain installed, rebuild-ccl will not work. See Section 3.3, “Kernel Build Prerequisites” for additional details.

Most distributions of Linux have all or most of the required development tools either pre-installed or readily available. On Debian-based Linux you can download and install the essential build tools using the package manager. For example:

apt-get install build-essential

(You may need to install C header files separately.)

For Mac OS X, Xcode 4 is available from the App Store.

For Windows, install Cygwin and the MinGW toolchain for the 32- or 64-bit OS. More information about installing Clozure CL on Windows is available in the Clozure CL Wiki at URL: http://trac.clozure.com/ccl/wiki/WindowsNotes

2.2.4.2. When to Rebuild Clozure CL From Sources

The most common scenario that requires a full rebuild is the standard installation after downloading the source tree. Users and application developers (who otherwise have no special build requirements) will generally need to run the full rebuild process just once for any given installation on a particular host system.

Another common scenario is installing a patch update: You can use Subversion (svn update) to download a more recent set of source files. (Be sure to download sources from the same path and branch in the source repository.) Then run a full rebuild to create new kernel and heap images. If you are running Clozure CL from the trunk, you may need to update sources and run the full rebuild more often.

Another reason to do a full rebuild is to ensure that Clozure CL will run properly in the host OS environment. This may be necessary, for example, when the target OS version is not identical to the one where the pre-built kernel was generated. The Lisp kernel uses some functionality defined in standard platform-provided libraries. On some platforms, applications (such as the Lisp kernel) are built in such a way as to depend on the specific versions of these libraries that were present at build time, and may not run on systems that have older or newer versions of these libraries. If you're affected by this, the simplest workaround is to build the Lisp kernel on the machine(s) that you intend to run it on and use that locally-built kernel instead of one distributed via Subversion.

2.2.4.3. Rebuilding Clozure CL Using REBUILD-CCL

Once the checkout is complete, and provided that you have a working development setup, you can build Clozure CL by running the Lisp kernel (an OS-native executable program) and running REBUILD-CCL in Lisp.

For example, to build a 64-bit Clozure CL on Mac OS X:

joe:ccl> ./dx86cl64 --no-init
Welcome to Clozure Common Lisp Version 1.7  (DarwinX8664)!
? (rebuild-ccl :full t)
Rebuilding Clozure Common Lisp using Version 1.7  (DarwinX8664)
;Building lisp-kernel ...
;Kernel built successfully.
;Compiling <...>
;Loading <...>

<...lots of compilation output...>

;Wrote bootstrapping image: #P"/Users/joe/ccl/x86-boot64.image"
;Wrote heap image: #P"/Users/joe/ccl/dx86cl64.image"
NIL
? (quit)
joe:ccl>
    

If the build fails for any reason, the kernel and/or heap image files may be missing or corrupted. To recover, delete the image files and update the source directory from Subversion. For example:

joe:ccl> rm dx86cl*
joe:ccl> svn update
<... lots of Subversion output...>
joe:ccl> ./dx86cl64 --no-init
Welcome to Clozure Common Lisp Version 1.7  (DarwinX8664)!
? (rebuild-ccl :full t)
<... lots of compilation output...>
? (quit)
joe:ccl> 

    

Once the full rebuild is completed, you can run the new Lisp kernel from the command shell. However, running the OS- and processor-specific executable directly is not recommended for day-to-day use. Clozure CL includes the ccl and ccl64 command shell scripts. For details on configuring a shell script for your environment, see Section 2.3.1, “The ccl Shell Script”.

2.2.4.4. Summary of the Build Process Steps

Should the build fail, your first concern should be to confirm that all requirements are in place: the C compiler, utilities, and OS header files; source files for the trunk or release branch you want to build; and the Lisp kernel and heap image files. For assistance with trouble-shooting, here is an outline of the full build process, with links to the more detailed instructions in Chapter 3, Building Clozure CL from its Source Code.

2.3. Command Line Set Up

Sometimes it's convenient to use Clozure CL from a Unix shell command line. This is especially true when using Clozure CL as a way to run Common Lisp utilities.

2.3.1. The ccl Shell Script

Clozure CL needs to be able to find the ccl directory in order to support features such as require and provide, access to foreign interface information (see The Interface Database) and the Lisp build process (see Building Clozure CL from its Source Code). Specifically, it needs to set up logical pathname translations for the "ccl:" logical host. If this logical host isn't defined (or isn't defined correctly), some things might work, some things might not, and it'll generally be hard to invoke and use Clozure CL productively.

Clozure CL uses the value of the environment variable CCL_DEFAULT_DIRECTORY to determine the filesystem location of the ccl directory; the ccl shell script is intended to provide a way to invoke Clozure CL with that environment variable set correctly.

There are two versions of the shell script: "ccl/scripts/ccl" is used to invoke 32-bit implementations of Clozure CL and "ccl/scripts/ccl64" is used to invoke 64-bit implementations. Install one script or the other or both as needed.

To use the script:

  1. Copy the script to a directory that is on your PATH. This is often /usr/local/bin or ~/bin. It is better to do this than to add ccl/scripts to your PATH, because the script needs to be edited, and editing it in-place means that Subversion sees the script as modified..

  2. Edit the definition of CCL_DEFAULT_DIRECTORY near the beginning of the shell script so that it refers to your ccl directory. Alternately, set the value of the CCL_DEFAULT_DIRECTORY environment variable wherever you usually set per-user environment variables, in your .cshrc, .tcshrc, .bashrc, .bash_profile, or .MacOSX/environment.plist script, or system-wide in /etc/profile or /etc/bashrc. When the ccl script runs, if the process environment contains a definition of CCL_DEFAULT_DIRECTORY, the ccl script will not override it.

  3. Ensure that the shell script is executable, for example:

    $ chmod +x ~/ccl/ccl/scripts/ccl64

    This command grants execute permission to the named script. If you are using a 32-bit platform, substitute "ccl" in place of "ccl64".

    Warning

    The above command won't work if you are not the owner of the installed copy of Clozure CL. In that case, you can use the "sudo" command like this:

    $ sudo chmod +x ~/ccl/ccl/scripts/ccl64

    Give your password when prompted.

    If the "sudo" command doesn't work, then you are not an administrator on the system you're using, and you don't have the appropriate "sudo" permissions. In that case you'll need to get help from the system's administrator.

Note that most people won't need both ccl and ccl64 scripts. You only need both if you sometimes run 32-bit Clozure CL and sometimes run 64-bit Clozure CL. You can rename the script that you use to whatever you want. For example, if you are on a 64-bit system, and you only use Clozure CL in 64-bit mode, then you can rename ccl64 to ccl so that you only need to type "ccl" to run it.

Once this is done, it should be possible to invoke Clozure CL by typing ccl or ccl64 at a shell prompt:

shell> ccl
Welcome to Clozure Common Lisp Version 1.7  (DarwinX8632)!
?
      

The ccl shell script passes all of its arguments to the Clozure CL kernel. See Section 2.3.2, “Invocation” for more information about command-line arguments.

Assuming the shell script is configured and invoked properly, Clozure CL should be able to initialize the "ccl:" logical host so that its translations refer to the "ccl" directory. To test this, you can call probe-file in Clozure CL's read-eval-print loop:

? (probe-file "ccl:level-1;level-1.lisp")  ;returns the physical pathname of the file
#P"/Users/joe/my_lisp_stuff/ccl/level-1/level-1.lisp"
      

2.3.2. Invocation

Assuming that the shell script is properly installed, it can be used to invoke Clozure CL from a shell prompt:

shell> ccl [args ...]

By convention ccl runs a 32-bit session; ccl64 runs a 64-bit session. However, the name of the installed script(s) and the implementation that is invoked are customizable, as described in Section 2.3.1, “The ccl Shell Script”.

For details about command-line options see Section 2.5, “Command Line Options”.

2.4. Personal Customization with the Init File

By default Clozure CL tries to load the file "home:ccl-init.lisp" or the compiled "home:ccl-init.fasl" upon starting up. Clozure CL does this by executing (load "home:ccl-init"). If it's unable to load the file (for example because the file doesn't exist), Clozure CL doesn't signal an error or warning, it just completes its startup normally.

On Unix systems, if "ccl-init.lisp" is not present, Clozure CL will look for ".ccl-init.lisp" (post 1.2 versions only).

The "home:" prefix to the filename is a Common Lisp logical host, which Clozure CL initializes to refer to your home directory. Clozure CL therefore looks for either of the files ~/ccl-init.lisp or ~/ccl-init.fasl.

Because the init file is loaded the same way as normal Lisp code is, you can put anything you want in it. For example, you can change the working directory, and load packages that you use frequently.

To suppress the loading of this init-file, invoke Clozure CL with the --no-init option.

2.5. Command Line Options

When using Clozure CL from the command line, the following options may be used to modify its behavior. The exact set of Clozure CL command-line arguments may vary per platform and slowly changes over time. The current set of command line options may be retrieved by using the --help option.

  • -h (or --help). Provides a definitive (if somewhat terse) summary of the command line options accepted by the Clozure CL implementation and then exits.

  • -V (or --version). Prints the version of Clozure CL then exits. The version string is the same value that is returned by LISP-IMPLEMENTATION-VERSION.

  • -K character-encoding-name (or --terminal-encoding character-encoding-name). Specifies the character encoding to use for *TERMINAL-IO* (see Section 4.5.4, “Character Encodings”). Specifically, the character-encoding-name string is uppercased and interned in the KEYWORD package. If an encoding named by that keyword exists, CCL:*TERMINAL-CHARACTER-ENCODING-NAME* is set to the name of that encoding. CCL:*TERMINAL-CHARACTER-ENCODING-NAME* defaults to NIL, which is a synonym for :ISO-8859-1.

    For example:

    shell> ccl -K utf-8
    	      

    has the effect of making the standard CL streams use :UTF-8 as their character encoding.

  • -n (or --no-init). If this option is given, the init file is not loaded. This is useful if Clozure CL is being invoked by a shell script that should not be affected by whatever customizations a user might have in place.

  • -e form (or --eval). An expression is read (via READ-FROM-STRING) from the string form and evaluated. If form contains shell metacharacters, it may be necessary to escape or quote them to prevent the shell from interpreting them.

  • -l path (or --load path). Loads file specified by path.

  • -T n (or --set-lisp-heap-gc-threshold n). Sets the Lisp gc threshold to n. (see Section 16.3, “GC Page reclamation policy”

  • -Q (or --quiet). Suppresses printing of heralds and prompts when the --batch command line option is specified.

  • -R n (or --heap-reserve). Reserves n bytes for heap expansion. The default is 549755813888. (see Section 16.1, “Heap space allocation”)

  • -S n (or --stack-size n). Sets the size of the initial control stack to n. (see Section 7.3.1, “Thread Stack Sizes”)

  • -Z n (or --thread-stack-size n). Sets the size of the first thread's stack to n. (see Section 7.3.1, “Thread Stack Sizes”)

  • -b (or --batch). Execute in "batch mode". End-of-file from *STANDARD-INPUT* causes Clozure CL to exit, as do attempts to enter a break loop.

  • --no-sigtrap An obscure option for running under GDB.

  • -I image-name (or --image-name image-name). Specifies the image name for the kernel to load. Defaults to the kernel name with ".image" appended.

The --load and --eval options can each be provided multiple times. They're executed in the order specified on the command line, after the init file (if there is one) is loaded and before the toplevel read-eval-print loop is entered.

Finally, any arguments following the pseudo-argument -- are not processed, and are made available to Lisp as the value of ccl:*unprocessed-command-line-arguments*.

2.6. Using Clozure CL with GNU Emacs and SLIME

SLIME (see the SLIME web page) is an Emacs mode for interacting with Common Lisp systems. Clozure CL is well-supported by SLIME.

See the InstallingSlime topic on the Clozure CL wiki for some tips on how to get SLIME running with Clozure CL.

2.7. Example Programs

A number (ok, a small number), of example programs are distributed in the "ccl:examples;" directory of the source distribution. See the README-OPENMCL-EXAMPLES text file in that directory for information about prerequisites and usage.

Some of the example programs are derived from C examples in textbooks, etc.; in those cases, the original author and work are cited in the source code.

Unless the original author or contributor claims other rights, you're free to incorporate any of this example code or derivative thereof in any of your own works without restriction. In doing so, you agree that the code was provided "as is", and that no other party is legally or otherwise responsible for any consequences of your decision to use it.

If you've developed Clozure CL examples that you'd like to see added to the distribution, please send mail to the Clozure CL mailing lists. Any such contributions would be welcome and appreciated (as would bug fixes and improvements to the existing examples.)

Chapter 3. Building Clozure CL from its Source Code

Clozure CL, like many other Lisp implementations, consists of a kernel and a heap image. The kernel is an ordinary C program, and is built with a C compiler. It provides very basic and fundamental facilities, such as memory management, garbage collection, and bootstrapping. All the higher-level features are written in Lisp, and compiled into the heap image. Both parts are needed to have a working Lisp implementation; neither the kernel nor the heap image can stand alone.

You may already know that, when you have a C compiler which is written in C, you need a working C compiler to build the compiler. Similarly, the Clozure CL heap image includes a Lisp compiler, which is written in Lisp. You therefore need a working Lisp compiler in order to build the Lisp heap image.

Where will you get a working Lisp compiler? No worries; you can use a precompiled copy of a (slightly older and compatible) version of Clozure CL. This section explains how to do all this.

In principle it should be possible to use another implementation of Common Lisp as the host compiler, rather than an older Clozure CL; this would be a challenging and experimental way to build, and is not described here.

3.1. Building Definitions

The following terms are used in subsequent sections; it may be helpful to refer to these definitions.

fasl files are the object files produced by compile-file. fasl files store the machine code associated with function definitions and the external representation of other lisp objects in a compact, machine-readable form. fasl is short for “FASt Loading”. Clozure CL uses different pathname types (extensions) to name fasl files on different platforms; see Table 3.1, “Platform-specific filename conventions”

The Lisp kernel is a C program with a fair amount of platform-specific assembly language code. Its basic job is to map a lisp heap image into memory, transfer control to some compiled lisp code that the image contains, handle any exceptions that occur during the execution of that lisp code, and provide various other forms of runtime support for that code. Clozure CL uses different filenames to name the lisp kernel files on different platforms; see Table 3.1, “Platform-specific filename conventions”.

A heap image is a file that can be quickly mapped into a process's address space. Conceptually, it's not too different from an executable file or shared library in the OS's native format (ELF or Mach-O/dyld format); for historical reasons, Clozure CL's own heap images are in their own (fairly simple) format. The term full heap image refers to a heap image file that contains all of the code and data that comprise Clozure CL. Clozure CL uses different filenames to name the standard full heap image files on different platforms; see Table 3.1, “Platform-specific filename conventions”.

A bootstrapping image is a minimal heap image used in the process of building Clozure CL itself. The bootstrapping image contains just enough code to load the rest of Clozure CL from fasl files. It may help to think of the bootstrapping image as the egg and the full heap image as the chicken. Clozure CL uses different filenames to name the standard bootstrapping image files on different platforms; see Table 3.1, “Platform-specific filename conventions” .

Each supported platform (and possibly a few as-yet-unsupported ones) has a uniquely named subdirectory of ccl/lisp-kernel/; each such contains a Makefile and may contain some auxiliary files (linker scripts, etc.) that are used to build the lisp kernel on a particular platform.The platform-specific name of the kernel build directory is described in Table 3.1, “Platform-specific filename conventions”.

3.1.1. Platform-specific filename conventions

Table 3.1. Platform-specific filename conventions

Platform kernel full-image boot-image fasl extension kernel-build directory
DarwinPPC32 dppccl dppccl.image ppc-boot.image .dfsl darwinppc
LinuxPPC32 ppccl ppccl.image ppc-boot .pfsl linuxppc
DarwinPPC64 dppccl64 dppccl64.image ppc-boot64.image .d64fsl darwinppc64
LinuxPPC64 ppccl64 ppccl64.image ppc-boot64 .p64fsl linuxppc64
LinuxX8664 lx86cl64 lx86cl64.image x86-boot64 .lx64fsl linuxx8664
LinuxX8632 lx86cl lx86cl.image x86-boot32 .lx32fsl linuxx8632
DarwinX8664 dx86cl64 dx86cl64.image x86-boot64.image .dx64fsl darwinx8664
DarwinX8632 dx86cl dx86cl.image x86-boot32.image .dx32fsl darwinx8632
FreeBSDX8664 fx86cl64 fx86cl64.image fx86-boot64 .fx64fsl freebsdx8664
FreeBSDX8632 fx86cl fx86cl.image fx86-boot32 .fx32fsl freebsdx8632
SolarisX64 sx86cl64 sx86cl64.image sx86-boot64 .sx64fsl solarisx64
SolarisX86 sx86cl sx86cl.image sx86-boot32 .sx32fsl solarisx86
Win64 wx86cl64.exe sx86cl64.image wx86-boot64.image .wx64fsl win64
Win32 wx86cl.exe wx86cl.image wx86-boot32.image .wx32fsl win32

3.2. Setting Up to Build

At a given time, there are generally two versions of Clozure CL that you might want to use (and therefore might want to build from source):

  • The released version

  • The development version, called the "trunk", which may contain both interesting new features and interesting new bugs

All versions are available for download from svn.clozure.com via the Subversion source control system.

For example, to get a released version (1.7 in this example), use a command like:

svn co http://svn.clozure.com/publicsvn/openmcl/release/1.7/xxx/ccl

To get the trunk version, use:

svn co http://svn.clozure.com/publicsvn/openmcl/trunk/xxx/ccl

Change the xxx to one of the following names: darwinx86, linuxx86, freebsdx86, solarisx86, windows, linuxppc, or darwinppc.

Tarball distributions of released versions are also available for download via ftp from: ftp://clozure.com/pub/release/. For additional information about availability of source and distributions see the Clozure CL Trac.

Subversion client programs are pre-installed on Mac OS X 10.5 and later and are typically either pre-installed or readily available on Linux and FreeBSD platforms. The Subversion web page contains links to Subversion client programs for many platforms. Users of Mac OS X 10.4 or later can also install Subversion clients via Fink or MacPorts. On Debian Linux (and on related Linux distros such as Ubuntu) run apt-get install subversion or equivalent in the command-line or interactive package manager.

3.3. Kernel Build Prerequisites

The Clozure CL kernel can be built with the following widely available tools:

  • cc or gcc — the GNU C compiler

  • ld — the GNU linker

  • m4 or gm4 — the GNU m4 macro processor

  • as — the GNU assembler (version 2.10.1 or later)

  • make — either GNU make or, on FreeBSD, the default BSD make program

In general, the more recent the versions of those tools, the better; some versions of gcc 3.x on Linux have difficulty compiling some of the kernel source code correctly (so gcc 4.0 should be used, if possible.) On Mac OS X, the versions of the tools distributed with Xcode should work fine; on Linux, the versions of the tools installed with the OS (or available through its package management system) should work fine if they're "recent enough". On FreeBSD, the installed version of the m4 program doesn't support some features that the kernel build process depends on; the GNU version of the m4 macroprocessor (called gm4 on FreeBSD) should be installed.

Note

In order to build the lisp kernel on Mac OS X 10.6 Snow Leopard, you must install the optional 10.4 support when installing Xcode.

3.4. Building Everything

You now have everything you need. Start up Clozure CL with the -n or --no-init option to avoid potential interference from code in your init file, and evaluate the following form to bring your Lisp system up to date.

? (ccl:rebuild-ccl :full t)
    

That call to the function rebuild-ccl performs the following steps:

  • Deletes all fasl files and other object files in the ccl directory tree

  • Runs an external process that does a make in the current platform's kernel build directory to create a new kernel. This step can only work if the C compiler and related tools are installed; see Section 3.3, “Kernel Build Prerequisites”.

  • Does (compile-ccl t) in the running lisp, to produce a set of fasl files from the “higher level” lisp sources.

  • Does (xload-level-0 :force) in the running lisp, to compile the lisp sources in the “ccl:level-0;” directory into fasl files and then create a bootstrapping image from those fasl files.

  • Runs another external process, which causes the newly compiled lisp kernel to load the new bootstrapping image. The bootstrapping image then loads the “higher level” fasl files and a new copy of the platform's full heap image is then saved.

If all goes well, it'll all happen without user intervention and with some simple progress messages. If anything goes wrong during execution of either of the external processes, the process output is displayed as part of a lisp error message.

rebuild-ccl is essentially just a short cut for running all the individual steps involved in rebuilding the system. You can also execute these steps individually, as described below.

3.5. Building the Kernel

The Lisp kernel is the executable that you run to use Lisp. It doesn't actually contain the entire Lisp implementation; rather, it loads a heap image which contains the specifics—the "library", as it might be called if this was a C program. The kernel also provides runtime support to the heap image, such as garbage collection, memory allocation, exception handling, and the OS interface.

The Lisp kernel file has different names on different platforms. See Table 3.1, “Platform-specific filename conventions”. On all platforms the lisp kernel sources reside in ccl/lisp-kernel.

This section gives directions on how to rebuild the Lisp kernel from its source code. Most Clozure CL users will rarely have to do this. You probably will only need to do it if you are attempting to port Clozure CL to a new architecture or extend or enhance its kernel in some way. As mentioned above, this step happens automatically when you do

? (rebuild-ccl :full t)
      

3.5.1. Using "make" to build the lisp kernel

With those tools in place, do:

shell> cd ccl/lisp-kernel/PLATFORM
shell> make
	    

That'll assemble several assembly language source files, compile several C source files, and link ../../the kernel.

3.6. Building the Heap Image

The initial heap image is loaded by the Lisp kernel, and provides most of the language implementation The heap image captures the entire state of a running Lisp (except for external resources, such as open files and TCP sockets). After it is loaded, the contents of the new Lisp process's memory are exactly the same as those of the old Lisp process when the image was created.

The heap image is how we get around the fact that we can't run Lisp code until we have a working Lisp implementation, and we can't make our Lisp implementation work until we can run Lisp code. Since the heap image already contains a fully-working implementation, all we need to do is load it into memory and start using it.

If you're building a new version of Clozure CL, you need to build a new heap image.

(You might also wish to build a heap image if you have a large program that is very complicated or time-consuming to load, so that you will be able to load it once, save an image, and thenceforth never have to load it again. At any time, a heap image capturing the entire memory state of a running Lisp can be created by calling the function ccl:save-application.)

3.6.1. Development cycle

Creating a new Clozure CL full heap image consists of the following steps:

  1. Using your existing Clozure CL, create a bootstrapping image

  2. Using your existing Clozure CL, recompile your updated Clozure CL sources

  3. Invoke Clozure CL with the bootstrapping image you just created (rather than with the existing full heap image).

When you invoke Clozure CL with the bootstrapping image, it starts up, loads all of the Clozure CL fasl files, and saves out a new full heap image. Voila. You've created a new heap image.

A few points worth noting:

  • There's a circular dependency between the full heap image and the bootstrapping image, in that each is used to build the other.

  • There are some minor implementation differences, but the environment in effect after the bootstrapping image has loaded its fasl files is essentially equivalent to the environment provided by the full heap image; the latter loads a lot faster and is easier to distribute, of course.

  • If the full heap image doesn't work (because of an OS compatibilty problem or other bug), it's very likely that the bootstrapping image will suffer the same problems.

Given a bootstrapping image and a set of up-to-date fasl files, the development cycle usually involves editing lisp sources (or updating those sources via svn update), recompiling modified files, and using the bootstrapping image to produce a new heap image.

3.6.2. Generating a bootstrapping image

The bootstrapping image isn't provided in Clozure CL distributions. It can be built from the source code provided in distributions (using a lisp image and kernel provided in those distributions) using the procedure described below.

The bootstrapping image is built by invoking a special utility inside a running Clozure CL heap image to load files contained in the ccl/level-0 directory. The bootstrapping image loads several dozen fasl files. After it's done so, it saves a heap image via save-application. This process is called "cross-dumping".

Given a source distribution, a lisp kernel, and a heap image, one can produce a bootstrapping image by first invoking Clozure CL from the shell:

shell> ccl
Welcome to Clozure CL .... !
?
	  

then calling ccl:xload-level-0 at the lisp prompt:

? (ccl:xload-level-0)
	  

This function compiles the lisp sources in the ccl/level-0 directory if they're newer than the corresponding fasl files and then loads the resulting fasl files into a simulated lisp heap contained in data structures inside the running lisp. That simulated heap image is then written to disk.

xload-level-0 should be called whenever your existing boot image is out-of-date with respect to the source files in ccl:level-0; — For example:

? (ccl:xload-level-0 :force)
      

forces recompilation of the level-0 sources.

3.6.3. Generating fasl files

Calling:

? (ccl:compile-ccl)
	  

at the lisp prompt compiles any fasl files that are out-of-date with respect to the corresponding lisp sources; (ccl:compile-ccl t) forces recompilation. ccl:compile-ccl reloads newly-compiled versions of some files; ccl:xcompile-ccl is analogous, but skips this reloading step.

Unless there are bootstrapping considerations involved, it usually doesn't matter whether these files are reloaded after they're recompiled.

Calling compile-ccl or xcompile-ccl in an environment where fasl files don't yet exist may produce warnings to that effect whenever files are required during compilation; those warnings can be safely ignored. Depending on the maturity of the Clozure CL release, calling compile-ccl or xcompile-ccl may also produce several warnings about undefined functions, etc. They should be cleaned up at some point.

3.6.4. Building a full image from a bootstrapping image

To build a full image from a bootstrapping image, just invoke the kernel with the bootstrapping image as an argument

$ cd ccl                        # wherever your ccl directory is
$ ./KERNEL --image-name BOOT_IMAGE --no-init
	  

Where KERNEL and BOOT_IMAGE are the names of the kernel and boot image appropriate to the platform you are running on. See Table 3.1, “Platform-specific filename conventions”

That should load a few dozen fasl files (printing a message as each file is loaded.) If all of these files successfully load, the lisp will print a prompt. You should be able to do essentially everything in that environment that you can in the environment provided by a "real" heap image. If you're confident that things loaded OK, you can save that image:

? (ccl:save-application "image_name") ; Overwriting the existing heap image
	  

Where image_name is the name of the full heap image for your platform. See Table 3.1, “Platform-specific filename conventions”.

If things go wrong in the early stages of the loading sequence, errors are often difficult to debug; until a fair amount of code (CLOS, the CL condition system, streams, the reader, the read-eval-print loop) is loaded, it's generally not possible for the lisp to report an error. Errors that occur during these early stages ("the cold load") sometimes cause the lisp kernel debugger (see ) to be invoked; it's primitive, but can sometimes help one to get oriented.

Chapter 4. Using Clozure CL

4.1. Introduction

The Common Lisp standard allows considerable latitude in the details of an implementation, and each particular Common Lisp system has some idiosyncrasies. This chapter describes ordinary user-level features of Clozure CL, including features that may be part of the Common Lisp standard, but which may have quirks or details in the Clozure CL implementation that are not described by the standard. It also describes extensions to the standard; that is, features of Clozure CL that are not part of the Common Lisp standard at all.

4.2. Trace

Clozure CL's tracing facility is invoked by an extended version of the Common Lisp trace macro. Extensions allow tracing of methods, as well as finer control over tracing actions.

TRACE {keyword global-value}* {spec | (spec {keyword local-value}*)}* [Macro]

The trace macro encapsulates the functions named by specs, causing trace actions to take place on entry and exit from each function. The default actions print a message on function entry and exit. Keyword/value options can be used to specify changes in the default behavior.

Invoking (trace) without arguments returns a list of functions being traced.

A spec is either a symbol that is the name of a function, or an expression of the form (setf symbol), or a specific method of a generic function in the form (:method gf-name {qualifier}* ({specializer}*)), where a specializer can be the name of a class or an EQL specializer.

A spec can also be a string naming a package, or equivalently a list (:package package-name), in order to request that all functions in the package to be traced.

By default, whenever a traced function is entered or exited, a short message is printed on *trace-output* showing the arguments on entry and values on exit. Options specified as key/value pairs can be used to modify this behavior. Options preceding the function specs apply to all the functions being traced. Options specified along with a spec apply to that spec only and override any global options. The following options are supported:

:methods {T | nil}

If true, and if applied to a spec naming a generic function, arranges to trace all the methods of the generic function in addition to the generic function itself.

:inside outside-spec | ({outside-spec}*)

Inhibits all trace actions unless the current invocation of the function being traced is inside one of the outside-spec's, i.e. unless a function named by one of the outside-spec's is currently on the stack. outside-spec can name a function, a method, or a package, as above.

:if form, :condition form

Evaluates form whenever the function being traced is about to be entered, and inhibits all trace actions if form returns nil. The form may reference the lexical variable ccl::args, which is a list of the arguments in this call. :condition is just a synonym for :if, though if both are specified, both must return non-nil.

:before-if form

Evaluates form whenever the function being traced is about to be entered, and inhibits the entry trace actions if form returns nil. The form may reference the lexical variable ccl::args, which is a list of the arguments in this call. If both :if and :before-if are specified, both must return non-nil in order for the before entry actions to happen.

:after-if form

Evaluates form whenever the function being traced has just exited, and inhibits the exit trace actions if form returns nil. The form may reference the lexical variable ccl::vals, which is a list of values returned by this call. If both :if and :after-if are specified, both must return non-nil in order for the after exit actions to happen.

:print-before form

Evaluates form whenever the function being traced is about to be entered, and prints the result before printing the standard entry message. The form may reference the lexical variable ccl::args, which is a list of the arguments in this call. To see multiple forms, use values: :print-before (values (one-thing) (another-thing)).

:print-after form

Evaluates form whenever the function being traced has just exited, and prints the result after printing the standard exit message. The form may reference the lexical variable ccl::vals, which is a list of values returned by this call. To see multiple forms, use values: :print-after (values (one-thing) (another-thing)).

:print form

Equivalent to :print-before form :print-after form.

:eval-before form

Evaluates form whenever the function being traced is about to be entered. The form may reference the lexical variable ccl::args, which is a list of the arguments in this call.

:eval-after form

Evaluates form whenever the function being has just exited. The form may reference the lexical variable ccl::vals, which is a list of values returned by this call.

:eval form

Equivalent to :eval-before form :eval-after form.

:break-before form

Evaluates form whenever the function being traced is about to be entered, and if the result is non-nil, enters a debugger break loop. The form may reference the lexical variable ccl::args, which is a list of the arguments in this call.

:break-after form

Evaluates form whenever the function being traced has just exited, and if the result is non-nil, enters a debugger break loop. The form may reference the lexical variable ccl::vals, which is a list of values returned by this call.

:break form

Equivalent to :break-before form :break-after form.

:backtrace-before form, :backtrace form

Evaluates form whenever the function being traced is about to be entered. The form may reference the lexical variable ccl::args, which is a list of the arguments in this call. The value returned by form is intepreted as follows:

nil

does nothing

:detailed

prints a detailed backtrace to *trace-output*.

(:detailed integer)

prints the top integer frames of detailed backtrace to *trace-output*.

integer

prints top integer frames of a terse backtrace to *trace-output*.

anything else

prints a terse backtrace to *trace-output*.

Note that unlike with the other options, :backtrace is equivalent to :backtrace-before only, not both before and after, since it's usually not helpful to print the same backtrace both before and after the function call.

:backtrace-after form

Evaluates form whenever the function being traced has just exited. The form may reference the lexical variable ccl::vals, which is a list of values returned by this call. The value returned by form is intepreted as follows:

nil

does nothing

:detailed

prints a detailed backtrace to *trace-output*.

(:detailed integer)

prints the top integer frames of detailed backtrace to *trace-output*.

integer

prints top integer frames of a terse backtrace to *trace-output*.

anything else

prints a terse backtrace to *trace-output*.

:before action

specifies the action to be taken just before the traced function is entered. action is one of:

:print

The default, prints a short indented message showing the function name and the invocation arguments

:break

Equivalent to :before :print :break-before t

:backtrace

Equivalent to :before :print :backtrace-before t

function

Any other value is interpreted as a function to call on entry instead of printing the standard entry message. It is called with its first argument being the name of the function being traced, the remaining arguments being all the arguments to the function being traced, and ccl:*trace-level* bound to the current nesting level of trace actions.

:after action

specifies the action to be taken just after the traced function exits. action is one of:

:print

The default, prints a short indented message showing the function name and the returned values

:break

Equivalent to :after :print :break-after t

:backtrace

Equivalent to :after :print :backtrace-after t

function

Any other value is interpreted as a function to call on exit instead of printing the standard exit message. It is called with its first argument being the name of the function being traced, the remaining arguments being all the values returned by the function being traced, and ccl:*trace-level* bound to the current nesting level of trace actions.

CCL:*TRACE-LEVEL* [Variable]

Variable bound to the current nesting level during execution of before and after trace actions. The default printing actions use it to determine the amount of indentation.

CCL:*TRACE-MAX-INDENT* [Variable]

The default before and after print actions will not indent by more than the value of ccl:*trace-max-indent* regardless of the current trace level.

CCL:TRACE-FUNCTION spec &key {keyword value}* [Function]

This is a functional version of the TRACE macro. spec and keywords are as for TRACE, except that all arguments are evaluated.

CCL:*TRACE-PRINT-LEVEL* [Variable]

The default print actions bind CL:*PRINT-LEVEL* to this value while printing. Note that this rebinding is only in effect during the default entry and exit messages. It does not apply to printing of :print-before/:print-after forms or any explicit printing done by user code.

CCL:*TRACE-PRINT-LENGTH* [Variable]

The default print actions bind CL:*PRINT-LENGTH* to this value while printing. Note that this rebinding is only in effect during the default entry and exit messages. It does not apply to printing of :print-before/:print-after forms or any explicit printing done by user code.

CCL:*TRACE-BAR-FREQUENCY* [Variable]

By default, this is nil. If non-nil it should be a integer, and the default entry and exit messages will print a | instead of space every this many levels of indentation.

4.3. Advising

The advise macro can be thought of as a more general version of trace. It allows code that you specify to run before, after, or around a given function, for the purpose of changing the behavior of the function. Each piece of added code is called a piece of advice. Each piece of advice has a unique name, so that you can have multiple pieces of advice on the same function, including multiple :before, :after, and :around pieces of advice.

The :name and :when keywords serve to identify the piece of advice. A later call to advise with the same values of :name and :when will replace the existing piece of advice; a call with different values will not.

[Macro]

advise spec form &key when name
Add a piece of advice to the function or method specified by spec according to form.

Arguments and Values:

spec--- A specification of the function on which to put the advice. This is either a symbol that is the name of a function or generic function, or an expression of the form (setf symbol), or a specific method of a generic function in the form (:method symbol {qualifiers} (specializer {specializer})).

form--- A form to execute before, after, or around the advised function. The form can refer to the variable arglist that is bound to the arguments with which the advised function was called. You can exit from form with (return).

name--- A name that identifies the piece of advice.

when--- An argument that specifies when the piece of advice is run. There are three allowable values. The default is :before, which specifies that form is executed before the advised function is called. Other possible values are :after, which specifies that form is executed after the advised function is called, and :around, which specifies that form is executed around the call to the advised function. Use (:do-it) within form to indicate invocation of the original definition.

Examples:

The function foo, already defined, does something with a list of numbers. The following code uses a piece of advice to make foo return zero if any of its arguments is not a number. Using :around advice, you can do the following:

(advise foo (if (some #'(lambda (n) (not (numberp n))) arglist)
	      0
	      (:do-it))
	:when :around :name :zero-if-not-nums)
	

To do the same thing using a :before piece of advice:

(advise foo (if (some #'(lambda (n) (not (numberp n))) arglist)
	      (return 0))
	:when :before :name :zero-if-not-nums)
	

[Macro]

unadvise spec &key when name
Remove the piece or pieces of advice matching spec, when, and name.

Description:

The unadvise macro removes the piece or pieces of advice matching spec, when, and name. When the value of spec is t and the values of when and name are nil, unadvise removes every piece of advice; when spec is t, the argument when is nil, and name is non-nil, unadvise removes all pieces of advice with the given name.

Arguments and Values:

The arguments have the same meaning as in advise.

[Macro]

advisedp spec &key when name
Return a list of the pieces of advice matching spec, when, and name.

Description:

The advisedp macro returns a list of existing pieces of advice that match spec, when, and name. When the value of spec is t and the values of when and name are nil, advisedp returns all existing pieces of advice.

Arguments and Values:

The arguments have the same meaning as in advise.

4.4. Directory

Clozure CL's DIRECTORY function accepts the following implementation-dependent keyword arguments:

:files boolean

If true, includes regular (non-directory) files in DIRECTORY's output. Defaults to T.

:directories boolean

If true, includes directories in DIRECTORY's output. Defaults to NIL.

:all boolean

If true, includes files and directories whose names start with a dot character in DIRECTORY's output. (Entries whose name is "." or ".." are never included.) Defaults to T.

:follow-links boolean

If true, includes the TRUENAMEs of symbolic or hard links in DIRECTORY's output; if false, includes the link filenames without attempting to resolve them. Defaults to T.

Note that legacy HFS alias files are treated as plain files.

4.5. Unicode

All characters and strings in Clozure CL fully support Unicode by using UTF-32. There is only one CHARACTER type and one STRING type in Clozure CL. There has been a lot of discussion about this decision which can be found by searching the openmcl-devel archives at http://clozure.com/pipermail/openmcl-devel/. Suffice it to say that we decided that the simplicity and speed advantages of only supporting UTF-32 outweigh the space disadvantage.

4.5.1. Characters

There is one CHARACTER type in Clozure CL. All CHARACTERs are BASE-CHARs. CHAR-CODE-LIMIT is now #x110000, which means that all Unicode characters can be directly represented. As of Unicode 5.0, only about 100,000 of 1,114,112 possible CHAR-CODEs are actually defined. The function CODE-CHAR knows that certain ranges of code values (notably #xd800-#xddff) will never be valid character codes and will return NIL for arguments in that range, but may return a non-NIL value (an undefined/non-standard CHARACTER object) for other unassigned code values.

Clozure CL supports character names of the form u+xxxx—where x is a sequence of one or more hex digits. The value of the hex digits denotes the code of the character. The + character is optional, so #\u+0020, #\U0020, and #\U+20 all refer to the #\Space character.

Characters with codes in the range #xa0-#x7ff also have symbolic names These are the names from the Unicode standard with spaces replaced by underscores. So #\Greek_Capital_Letter_Epsilon can be used to refer to the character whose CHAR-CODE is #x395. To see the complete list of supported character names, look just below the definition for register-character-name in ccl:level-1;l1-reader.lisp.

4.5.2. External Formats

OPEN, LOAD, and COMPILE-FILE all take an :EXTERNAL-FORMAT keyword argument. The value of :EXTERNAL-FORMAT can be :DEFAULT (the default value), a line termination keyword (see Section 4.5.3, “Line Termination Keywords”), a character encoding keyword (see Section 4.5.4, “Character Encodings”), an external-format object created using CCL::MAKE-EXTERNAL-FORMAT (see make-external-format), or a plist with keys: :DOMAIN, :CHARACTER-ENCODING and :LINE-TERMINATION. If argument is a plist, the result of (APPLY #'MAKE-EXTERNAL-FORMAT argument) will be used.

If :DEFAULT is specified, then the value of CCL:*DEFAULT-EXTERNAL-FORMAT* is used. If no line-termination is specified, then the value of CCL:*DEFAULT-LINE-TERMINATION* is used, which defaults to :UNIX. If no character encoding is specified, then CCL:*DEFAULT-FILE-CHARACTER-ENCODING* is used for file streams and CCL:*DEFAULT-SOCKET-CHARACTER-ENCODING* is used for socket streams. The default, default character encoding is NIL which is a synonym for :ISO-8859-1.

Note that the set of keywords used to denote CHARACTER-ENCODINGs and the set of keywords used to denote line-termination conventions is disjoint: a keyword denotes at most a character encoding or a line termination convention, but never both.

EXTERNAL-FORMATs are objects (structures) with two read-only fields that can be accessed via the functions: EXTERNAL-FORMAT-LINE-TERMINATION and EXTERNAL-FORMAT-CHARACTER-ENCODING.

[Variable]

CCL:*DEFAULT-EXTERNAL-FORMAT*

Description:

The value of this variable is used when :EXTERNAL-FORMAT is unspecified or specified as :DEFAULT. It can meaningfully be given any value that can be used as an external-format (except for the value :DEFAULT.)

The initial value of this variable in Clozure CL is :UNIX, which is equivalent to (:LINE-TERMINATION :UNIX), among other things.

[Variable]

CCL:*DEFAULT-LINE-TERMINATION*

Description:

The value of this variable is used when an external-format doesn't specify a line-termination convention (or specifies it as :DEFAULT.) It can meaningfully be given any value that can be used as a line termination keyword (see Section 4.5.3, “Line Termination Keywords”).

The initial value of this variable in Clozure CL is :UNIX.

[Function]

make-external-format &key domain character-encoding line-termination => external-format
Either creates a new external format object, or return an existing one with the same specified slot values.

Arguments and Values:

domain---This is used to indicate where the external format is to be used. Its value can be almost anything. It defaults to NIL. There are two domains that have a pre-defined meaning in Clozure CL: :FILE indicates encoding for a file in the file system and :SOCKET indicates i/o to/from a socket. The value of domain affects the default values for character-encoding and line-termination.

character-encoding---A keyword that specifies the character encoding for the external format. Section 4.5.4, “Character Encodings”. Defaults to :DEFAULT which means if domain is :FILE use the value of the variable CCL:*DEFAULT-FILE-CHARACTER-ENCODING* and if domain is :SOCKET, use the value of the variable CCL:*DEFAULT-SOCKET-CHARACTER-ENCODING*. The initial value of both of these variables is NIL, which means the :ISO-8859-1 encoding.

line-termination---A keyword that indicates a line termination keyword Section 4.5.3, “Line Termination Keywords”. Defaults to :DEFAULT which means use the value of the variable CCL:*DEFAULT-LINE-TERMINATION*.

external-format---An external-format object as described above.

Description:

Despite the function's name, it doesn't necessarily create a new, unique EXTERNAL-FORMAT object: two calls to MAKE-EXTERNAL-FORMAT with the same arguments made in the same dynamic environment return the same (eq) object.

4.5.3. Line Termination Keywords

Line termination keywords indicate which characters are used to indicate the end of a line. On input, the external line termination characters are replaced by #\Newline and on output, #\Newlines are converted to the external line termination characters.

Table 4.1. Line Termination Keywords

keyword character(s)
:UNIX #\Linefeed
:MACOS #\Return
:CR #\Return
:CRLF #\Return #\Linefeed
:CP/M #\Return #\Linefeed
:MSDOS #\Return #\Linefeed
:DOS #\Return #\Linefeed
:WINDOWS #\Return #\Linefeed
:INFERRED see below
:UNICODE #\Line_Separator

:INFERRED means that a stream's line-termination convention is determined by looking at the contents of a file. It is only useful for FILE-STREAMs that're open for :INPUT or :IO. The first buffer full of data is examined, and if a #\Return character occurs before any #\Linefeed character, then the line termination type is set to :WINDOWS if that #\Return character is immediately followed by a #\Linefeed character and to :MACOS otherwise. If a #\Return character isn't found in the buffer or if #\Return is preceded by #\Linefeed, the file's line terminationt type is set to :UNIX.

4.5.4. Character Encodings

Internally, all characters and strings in Clozure CL are in UTF-32. Externally, files or socket streams may encode characters in a wide variety of ways. The International Organization for Standardization, widely known as ISO, defines many of these character encodings. Clozure CL implements some of these encodings as detailed below. These encodings are part of the specification of external formats Section 4.5.2, “External Formats”. When reading from a stream, characters are converted from the specified external character encoding to UTF-32. When writing to a stream, characters are converted from UTF-32 to the specified character encoding.

Internally, CHARACTER-ENCODINGs are objects (structures) that are named by character encoding keywords (:ISO-8859-1, :UTF-8, etc.). The structures contain attributes of the encoding and functions used to encode/decode external data, but unless you're trying to define or debug an encoding there's little reason to know much about the CHARACTER-ENCODING objects and it's usually preferable to refer to a character encoding by its name.

4.5.4.1. Encoding Problems

On output to streams with character encodings that can encode the full range of Unicode—and on input from any stream—"unencodable characters" are represented using the Unicode #\Replacement_Character (= #\U+fffd); the presence of such a character usually indicates that something got lost in translation. Either data wasn't encoded properly or there was a bug in the decoding process.

4.5.4.2. Byte Order Marks

The endianness of a character encoding is sometimes explicit, and sometimes not. For example, :UTF-16BE indicates big-endian, but :UTF-16 does not specify endianness. A byte order mark is a special character that may appear at the beginning of a stream of encoded characters to specify the endianness of a multi-byte character encoding. (It may also be used with UTF-8 character encodings, where it is simply used to indicate that the encoding is UTF-8.)

Clozure CL writes a byte order mark as the first character of a file or socket stream when the endianness of the character encoding is not explicit. Clozure CL also expects a byte order mark on input from streams where the endianness is not explicit. If a byte order mark is missing from input data, that data is assumed to be in big-endian order.

A byte order mark from a UTF-8 encoded input stream is not treated specially and just appears as a normal character from the input stream. It is probably a good idea to skip over this character.

4.5.4.3. DESCRIBE-CHARACTER-ENCODINGS

The set of character encodings supported by Clozure CL can be retrieved by calling CCL:DESCRIBE-CHARACTER-ENCODINGS.

[Function]

describe-character-encodings
Writes descriptions of defined character encodings to *terminal-io*.

Description:

Writes descriptions of all defined character encodings to *terminal-io*. These descriptions include the names of the encoding's aliases and a doc string which briefly describes each encoding's properties and intended use.

4.5.4.4. Supported Character Encodings

The list of supported encodings is reproduced here. Most encodings have aliases, e.g. the encoding named :ISO-8859-1 can also be referred to by the names :LATIN1 and :IBM819, among others. Where possible, the keywordized name of an encoding is equivalent to the preferred MIME charset name (and the aliases are all registered IANA charset names.)

:ISO-8859-1

An 8-bit, fixed-width character encoding in which all character codes map to their Unicode equivalents. Intended to support most characters used in most Western European languages.

Clozure CL uses ISO-8859-1 encoding for *TERMINAL-IO* and for all streams whose EXTERNAL-FORMAT isn't explicitly specified. The default for *TERMINAL-IO* can be set via the -K command-line argument (see Section 2.5, “Command Line Options”).

ISO-8859-1 just covers the first 256 Unicode code points, where the first 128 code points are equivalent to US-ASCII. That should be pretty much equivalent to what earliers versions of Clozure CL did that only supported 8-bit characters, but it may not be optimal for users working in a particular locale.

Aliases: :ISO_8859-1, :LATIN1, :L1, :IBM819, :CP819, :CSISOLATIN1

:ISO-8859-2

An 8-bit, fixed-width character encoding in which codes #x00-#x9f map to their Unicode equivalents and other codes map to other Unicode character values. Intended to provide most characters found in most languages used in Central/Eastern Europe.

Aliases: :ISO_8859-2, :LATIN2, :L2, :CSISOLATIN2

:ISO-8859-3

An 8-bit, fixed-width character encoding in which codes #x00-#x9f map to their Unicode equivalents and other codes map to other Unicode character values. Intended to provide most characters found in most languages used in Southern Europe.

Aliases: :ISO_8859-3, :LATIN3 :L3, :CSISOLATIN3

:ISO-8859-4

An 8-bit, fixed-width character encoding in which codes #x00-#x9f map to their Unicode equivalents and other codes map to other Unicode character values. Intended to provide most characters found in most languages used in Northern Europe.

Aliases: :ISO_8859-4, :LATIN4, :L4, :CSISOLATIN4

:ISO-8859-5

An 8-bit, fixed-width character encoding in which codes #x00-#x9f map to their Unicode equivalents and other codes map to other Unicode character values. Intended to provide most characters found in the Cyrillic alphabet.

Aliases: :ISO_8859-5, :CYRILLIC, :CSISOLATINCYRILLIC, :ISO-IR-144

:ISO-8859-6

An 8-bit, fixed-width character encoding in which codes #x00-#x9f map to their Unicode equivalents and other codes map to other Unicode character values. Intended to provide most characters found in the Arabic alphabet.

Aliases: :ISO_8859-6, :ARABIC, :CSISOLATINARABIC, :ISO-IR-127

:ISO-8859-7

An 8-bit, fixed-width character encoding in which codes #x00-#x9f map to their Unicode equivalents and other codes map to other Unicode character values. Intended to provide most characters found in the Greek alphabet.

Aliases: :ISO_8859-7, :GREEK, :GREEK8, :CSISOLATINGREEK, :ISO-IR-126, :ELOT_928, :ECMA-118

:ISO-8859-8

An 8-bit, fixed-width character encoding in which codes #x00-#x9f map to their Unicode equivalents and other codes map to other Unicode character values. Intended to provide most characters found in the Hebrew alphabet.

Aliases: :ISO_8859-8, :HEBREW, :CSISOLATINHEBREW, :ISO-IR-138

:ISO-8859-9

An 8-bit, fixed-width character encoding in which codes #x00-#xcf map to their Unicode equivalents and other codes map to other Unicode character values. Intended to provide most characters found in the Turkish alphabet.

Aliases: :ISO_8859-9, :LATIN5, :CSISOLATIN5, :ISO-IR-148

:ISO-8859-10

An 8-bit, fixed-width character encoding in which codes #x00-#x9f map to their Unicode equivalents and other codes map to other Unicode character values. Intended to provide most characters found in Nordic alphabets.

Aliases: :ISO_8859-10, :LATIN6, :CSISOLATIN6, :ISO-IR-157

:ISO-8859-11

An 8-bit, fixed-width character encoding in which codes #x00-#x9f map to their Unicode equivalents and other codes map to other Unicode character values. Intended to provide most characters found the Thai alphabet.

:ISO-8859-13

An 8-bit, fixed-width character encoding in which codes #x00-#x9f map to their Unicode equivalents and other codes map to other Unicode character values. Intended to provide most characters found in Baltic alphabets.

:ISO-8859-14

An 8-bit, fixed-width character encoding in which codes #x00-#x9f map to their Unicode equivalents and other codes map to other Unicode character values. Intended to provide most characters found in Celtic languages.

Aliases: :ISO_8859-14, :ISO-IR-199, :LATIN8, :L8, :ISO-CELTIC

:ISO-8859-15

An 8-bit, fixed-width character encoding in which codes #x00-#x9f map to their Unicode equivalents and other codes map to other Unicode character values. Intended to provide most characters found in Western European languages (including the Euro sign and some other characters missing from ISO-8859-1.

Aliases: :ISO_8859-15, :LATIN9

:ISO-8859-16

An 8-bit, fixed-width character encoding in which codes #x00-#x9f map to their Unicode equivalents and other codes map to other Unicode character values. Intended to provide most characters found in Southeast European languages.

Aliases: :ISO_8859-16, :ISO-IR-199, :LATIN8, :L8, :ISO-CELTIC

:MACINTOSH

An 8-bit, fixed-width character encoding in which codes #x00-#x7f map to their Unicode equivalents and other codes map to other Unicode character values. Traditionally used on Classic MacOS to encode characters used in western languages.

Aliases: :MACOS-ROMAN, :MACOSROMAN, :MAC-ROMAN, :MACROMAN

:UCS-2

A 16-bit, fixed-length encoding in which characters with CHAR-CODEs less than #x10000 can be encoded in a single 16-bit word. The endianness of the encoded data is indicated by the endianness of a byte-order-mark character (#u+feff) prepended to the data; in the absence of such a character on input, the data is assumed to be in big-endian order.

:UCS-2BE

A 16-bit, fixed-length encoding in which characters with CHAR-CODEs less than #x10000 can be encoded in a single 16-bit big-endian word. The encoded data is implicitly big-endian; byte-order-mark characters are not interpreted on input or prepended to output.

:UCS-2LE

A 16-bit, fixed-length encoding in which characters with CHAR-CODEs less than #x10000 can be encoded in a single 16-bit little-endian word. The encoded data is implicitly little-endian; byte-order-mark characters are not interpreted on input or prepended to output.

:US-ASCII

An 7-bit, fixed-width character encoding in which all character codes map to their Unicode equivalents.

Aliases: :CSASCII, :CP637, :IBM637, :US, :ISO646-US, :ASCII, :ISO-IR-6

:UTF-16

A 16-bit, variable-length encoding in which characters with CHAR-CODEs less than #x10000 can be encoded in a single 16-bit word and characters with larger codes can be encoded in a pair of 16-bit words. The endianness of the encoded data is indicated by the endianness of a byte-order-mark character (#u+feff) prepended to the data; in the absence of such a character on input, the data is assumed to be in big-endian order. Output is written in native byte-order with a leading byte-order mark.

:UTF-16BE

A 16-bit, variable-length encoding in which characters with CHAR-CODEs less than #x10000 can be encoded in a single 16-bit big-endian word and characters with larger codes can be encoded in a pair of 16-bit big-endian words. The endianness of the encoded data is implicit in the encoding; byte-order-mark characters are not interpreted on input or prepended to output.

:UTF-16LE

A 16-bit, variable-length encoding in which characters with CHAR-CODEs less than #x10000 can be encoded in a single 16-bit little-endian word and characters with larger codes can be encoded in a pair of 16-bit little-endian words. The endianness of the encoded data is implicit in the encoding; byte-order-mark characters are not interpreted on input or prepended to output.

:UTF-32

A 32-bit, fixed-length encoding in which all Unicode characters can be encoded in a single 32-bit word. The endianness of the encoded data is indicated by the endianness of a byte-order-mark character (#u+feff) prepended to the data; in the absence of such a character on input, input data is assumed to be in big-endian order. Output is written in native byte order with a leading byte-order mark.

Alias: :UTF-4

:UTF-32BE

A 32-bit, fixed-length encoding in which all Unicode characters encoded in a single 32-bit word. The encoded data is implicitly big-endian; byte-order-mark characters are not interpreted on input or prepended to output.

Alias: :UCS-4BE

:UTF-8

An 8-bit, variable-length character encoding in which characters with CHAR-CODEs in the range #x00-#x7f can be encoded in a single octet; characters with larger code values can be encoded in 2 to 4 bytes.

:UTF-32LE

A 32-bit, fixed-length encoding in which all Unicode characters can encoded in a single 32-bit word. The encoded data is implicitly little-endian; byte-order-mark characters are not interpreted on input or prepended to output.

Alias: :UCS-4LE

:Windows-31j

An 8-bit, variable-length character encoding in which character code points in the range #x00-#x7f can be encoded in a single octet; characters with larger code values can be encoded in 2 bytes.

Aliases: :CP932, :CSWINDOWS31J

:EUC-JP

An 8-bit, variable-length character encoding in which character code points in the range #x00-#x7f can be encoded in a single octet; characters with larger code values can be encoded in 2 bytes.

Alias: :EUCJP

:GB2312

An 8-bit, variable-length character encoding in which character code points in the range #x00-#x80 can be encoded in a single octet; characters with larger code values can be encoded in 2 bytes.

Alias: :GB2312-80 :GB2312-1980 :EUC-CN :EUCCN

:CP936

An 8-bit, variable-length character encoding in which character code points in the range #x00-#x80 can be encoded in a single octet; characters with larger code values can be encoded in 2 bytes.

Alias: :GBK :MS936 :WINDOWS-936

4.5.4.5. Encoding and Decoding Strings

Clozure CL provides functions to encode and decode strings to and from vectors of type (simple-array (unsigned-byte 8)).

[Function]

count-characters-in-octet-vector vector &key start end external-format

Description:

Returns the number of characters that would be produced by decoding vector (or the subsequence thereof delimited by start and end) according to external-format.

[Function]

decode-string-from-octets vector &key start end external-format string

Description:

Decodes the octets in vector (or the subsequence of it delimited by start and end) into a string according to external-format.

If string is supplied, output will be written into it. It must be large enough to hold the decoded characters. If string is not supplied, a new string will be allocated to hold the decoded characters.

Returns, as multiple values, the decoded string and the position in vector where the decoding ended.

Sequences of octets in vector that cannot be decoded into characters according to external-format will be decoded as #\Replacement_Character.

[Function]

encode-string-to-octets string &key start end external-format use-byte-order-mark vector vector-offset

Description:

Encodes string (or the substring delimited by start and end) into external-format and returns, as multiple values, a vector of octets containing the encoded data and an integer that specifies the offset into the vector where the encoded data ends.

When use-byte-order-mark is true, a byte-order mark will be included in the encoded data.

If vector is supplied, output will be written to it. It must be of type (simple-array (unsigned-byte 8)) and be large enough to hold the encoded data. If it is not supplied, the function will allocate a new vector.

If vector-offset is supplied, data will be written into the output vector starting at that offset.

Characters in string that cannot be encoded into external-format will be replaced with an encoding-dependent replacement character (#\Replacement_Character or #\Sub) before being encoded and written into the output vector.

[Function]

string-size-in-octets string &key start end external-format use-byte-order-mark

Description:

Returns the number of octets required to encode string (or the substring delimited by start and end) into external-format.

When use-byte-order-mark is true, the returned size will include space for a byte-order marker.

4.6. Pathnames

4.6.1. Pathname Expansion

Leading tilde (~) characters in physical pathname namestrings are expanded in the way that most shells do:

"~user/..." can be used to refer to an absolute pathname rooted at the home directory of the user named "user".

"~/..." can be used to refer to an absolute pathname rooted at the home directory of the current user.

4.6.2. Predefined Logical Hosts

Clozure CL sets up logical pathname translations for logical hosts: ccl and home

The CCL logical host should point to the ccl directory. It is used for a variety of purposes by Clozure CL including: locating Clozure CL source code, require and provide, accessing foreign function information, and the Clozure CL build process. It is set to the value of the environment variable CCL_DEFAULT_DIRECTORY, which is set by the openmcl shell script Section 2.3.1, “The ccl Shell Script”. If CCL_DEFAULT_DIRECTORY is not set, then it is set to the directory containing the current heap image.

4.6.3. Pathname Namestrings

The syntax of namestrings is implementation-defined in Common Lisp. Portable programs cannot assume much of anything about them. (See section 19.1.1 of the Common Lisp standard for more information.)

When translating a namestring into a pathname object, most implementations seem to follow the convention that a dot character in the namestring separates the pathname-name and the pathname-type. When there is more than one dot in involved, or when dots appear at the beginning or end of the namestrings, what to do is less clear: does ".emacs" describe a pathname whose name is nil and whose type is emacs or something else? Similarly, given "a.b.c", the question is which parts are parsed as the pathname name, and which are parsed as the pathname type?

When generating a namestring from a pathname object (as happens, for example, when printing a pathname), Clozure CL tries to avoid some potential ambiguity by escaping characters that might otherwise be used to separate pathname components. The character used to quote or escape the separators is a backlash on Unix systems, and a #\> character on Windows. So, for example, "a\\.b.c" has name "a.b" and type "c", whereas "a.b\\.c" has name "a" and type "b.c".

To get a native namestring suitable for passing to an operating system command, use the function ccl:native-translated-namestring.

4.6.3.1. Working with native namestrings

[Function]

native-translated-namestring pathname-designator
Return a namestring that uses the conventions of the native operating system.

Description:

This function returns a namestring that represents a pathname using the native conventions of the operating system. Any quoting or escaping of special characters will be removed.

For example, suppose that p is a pathname made by (make-pathname :name "a.b" :type "c"). Then, (native-translated-namestring p) evaluates to "a.b.c". By contrast, (namestring p) evaluates to "a\\.b.c".

[Macro]

with-filename-cstrs ( {(var value)}* ) {form}*
Suitably encode strings to be used as filenames for foreign code.

Description:

Executes forms in an environemt in which each var is bound to a stack-allocated foreign pointer which refers to a C-style string suitable for passing to foreign code which expects a filename argument.

For example, one might use this macro in the following way:

(with-filename-cstrs ((s (native-translated-namestring pathname)))
  (#_unlink s))
  

Various operating systems have different conventions for how they expect native pathname strings to be encoded. Darwin expects then to be decomposed UTF-8. The Unicode variants to Windows file-handling functions expect UTF-16. Other systems just treat them as opaque byte sequences. This macro ensures that the correct encoding is used, whatever the host operating system.

4.6.4. OS X (Darwin)

Clozure CL assumes that pathname strings are decomposed UTF-8.

4.6.5. Linux

Pathname strings are treated as null-terminated strings coded in the encoding named by the value returned by the function CCL:PATHNAME-ENCODING-NAME. This value may be changed with SETF.

4.6.6. FreeBSD

Pathname strings are treated as null-terminated strings encoded according to the current locale; a future release may change this convention to use UTF-8.

4.7. Memory-mapped Files

In release 1.2 and later, Clozure CL supports memory-mapped files. On operating systems that support memory-mapped files (including Mac OS X, Linux, and FreeBSD), the operating system can arrange for a range of virtual memory addresses to refer to the contents of an open file. As long as the file remains open, programs can read values from the file by reading addresses in the mapped range.

Using memory-mapped files may in some cases be more efficient than reading the contents of a file into a data structure in memory.

Clozure CL provides the functions CCL:MAP-FILE-TO-IVECTOR and CCL:MAP-FILE-TO-OCTET-VECTOR to support memory-mapping. These functions return vectors whose contents are the contents of memory-mapped files. Reading an element of such a vector returns data from the corresponding position in the file.

Without memory-mapped files, a common idiom for reading the contents of files might be something like this:

(let* ((stream (open pathname :direction :input :element-type '(unsigned-byte 8)))
       (vector (make-array (file-size-to-vector-size stream)
                           :element-type '(unsigned-byte 8))))
  (read-sequence vector stream))
    

Using a memory-mapped files has a result that is the same in that, like the above example, it returns a vector whose contents are the same as the contents of the file. It differs in that the above example creates a new vector in memory and copies the file's contents into it; using a memory-mapped file instead arranges for the vector's elements to point to the file's contents on disk directly, without copying them into memory first.

The vectors returned by CCL:MAP-FILE-TO-IVECTOR and CCL:MAP-FILE-TO-OCTET-VECTOR are read-only; any attempt to change an element of a vector returned by these functions results in a memory-access error. Clozure CL does not currently support writing data to memory-mapped files.

Vectors created by CCL:MAP-FILE-TO-IVECTOR and CCL:MAP-FILE-TO-OCTET-VECTOR are required to respect Clozure CL's limit on the total size of an array. That means that you cannot use these functions to create a vector longer than ARRAY-TOTAL-SIZE-LIMIT, even if the filesystem supports file sizes that are larger. The value of ARRAY-TOTAL-SIZE-LIMIT is (EXPT 2 24) on 32-but platforms; and (EXPT 2 56) on 64-bit platforms.

CCL:MAP-FILE-TO-IVECTOR pathname element-type [Function]

pathname

The pathname of the file to be memory-mapped.

element-type

The element-type of the vector to be created. Specified as a type-specifier that names a subtype of either SIGNED-BYTE or UNSIGNED-BYTE.

The map-file-to-ivector function tries to open the file at pathname for reading. If successful, the function maps the file's contents to a range of virtual addresses. If successful, it returns a read-only vector whose element-type is given by element-type, and whose contents are the contents of the memory-mapped file.

The returned vector is a displaced-array whose element-type is (UPGRADED-ARRAY-ELEMENT-TYPE element-type). The target of the displaced array is a vector of type (SIMPLE-ARRAY element-type (*)) whose elements are the contents of the memory-mapped file.

Because of alignment issues, the mapped file's contents start a few bytes (4 bytes on 32-bit platforms, 8 bytes on 64-bit platforms) into the vector. The displaced array returned by CCL:MAP-FILE-TO-IVECTOR hides this overhead, but it's usually more efficient to operate on the underlying simple 1-dimensional array. Given a displaced array (like the value returned by CCL:MAP-FILE-TO-IVECTOR), the function ARRAY-DISPLACEMENT returns the underlying array and the displacement index in elements.

Currently, Clozure CL supports only read operations on memory-mapped files. If you try to change the contents of an array returned by map-file-to-ivector, Clozure CL signals a memory error.

CCL:UNMAP-IVECTOR displaced-array [Function]

If the argument is a displaced-array returned by map-file-to-ivector, and if it has not yet been unmapped by this function, then unmap-ivector undoes the memory mapping, closes the mapped file, and changes the displaced-array so that its target is an empty vector (of length zero).

CCL:MAP-FILE-TO-OCTET-VECTOR pathname [Function]

This function is a synonym for (CCL:MAP-FILE-TO-IVECTOR pathname '(UNSIGNED-BYTE 8)) It is provided as a convenience for the common case of memory-mapping a file as a vector of bytes.

CCL:UNMAP-OCTET-VECTOR displaced-array [Function]

This function is a synonym for (CCL:UNMAP-IVECTOR)

4.8. Static Variables

Clozure CL supports the definition of static variables, whose values are the same across threads, and which may not be dynamically bound. The value of a static variable is thus the same across all threads; changing the value in one thread changes it for all threads.

Attempting to dynamically rebind a static variable (for instance, by using LET, or using the variable name as a parameter in a LAMBDA form) signals an error. Static variables are shared global resources; a dynamic binding is private to a single thread.

Static variables therefore provide a simple way to share mutable state across threads. They also provide a simple way to introduce race conditions and obscure bugs into your code, since every thread reads and writes the same instance of a given static variable. You must take care, therefore, in how you change the values of static variables, and use normal multithreaded programming techniques, such as locks or semaphores, to protect against race conditions.

In Clozure CL, access to a static variable is usually faster than access to a special variable that has not been declared static.

DEFSTATIC var value &key doc-string [Macro]

var

The name of the new static variable.

value

The initial value of the new static variable.

doc-string

A documentation string that is assigned to the new variable.

Proclaims the variable special, assigns the variable the supplied value, and assigns the doc-string to the variable's VARIABLE documentation. Marks the variable static, preventing any attempt to dynamically rebind it. Any attempt to dynamically rebind var signals an error.

4.9. Saving Applications

Clozure CL provides the function CCL:SAVE-APPLICATION, which creates a file containing an archived Lisp memory image.

Clozure CL consists of a small executable called the Lisp kernel, which implements the very lowest level features of the Lisp system, and an image, which contains the in-memory representation of most of the Lisp system, including functions, data structures, variables, and so on. When you start Clozure CL, you are launching the kernel, which then locates and reads an image file, restoring the archived image in memory. Once the image is fully restored, the Lisp system is running.

Using CCL:SAVE-APPLICATION, you can create a file that contains a modified image, one that includes any changes you've made to the running Lisp system. If you later pass your image file to the Clozure CL kernel as a command-line parameter, it then loads your image file instead of its default one, and Clozure CL starts up with your modifications.

If this scenario seems to you like a convenient way to create an application, that's just as intended. You can create an application by modifying the running Lisp until it does what you want, then use CCL:SAVE-APPLICATION to preserve your changes and later load them for use.

In fact, you can go further than that. You can replace Clozure CL's toplevel function with your own, and then, when the image is loaded, the Lisp system immediately performs your tasks rather than the default tasks that make it a Lisp development system. If you save an image in which you have done this, the resulting Lisp system is your tool rather than a Lisp development system.

You can go a step further still. You can tell CCL:SAVE-APPLICATION to prepend the Lisp kernel to the image file. Doing this makes the resulting image into a self-contained executable binary. When you run the resulting file, the Lisp kernel immediately loads the attached image file and runs your saved system. The Lisp system that starts up can have any behavior you choose. It can be a Lisp development system, but with your customizations; or it can immediately perform some task of your design, making it a specialized tool rather than a general development system.

In other words, you can develop any application you like by interactively modifying Clozure CL until it does what you want, then using CCL:SAVE-APPLICATION to preserve your changes in an executable image.

On Mac OS X, the application builder uses CCL:SAVE-APPLICATION to create the executable portion of the application bundle. Double-clicking the application bundle runs the executable image created by CCL:SAVE-APPLICATION.

Also on Mac OS X, Clozure CL supports an object type called MACPTR, which is the type of pointers into the foreign (Mac OS) heap. Examples of commonly-user MACPTR objects are Cocoa windows and other dynamically-allocated Mac OS system objects.

Because a MACPTR object is a pointer into a foreign heap that exists for the lifetime of the running Lisp process, and because a saved image is used by loading it into a brand new Lisp process, saved MACPTR objects cannot be relied on to point to the same things when reconstituted from a saved image. In fact, a restored MACPTR object might point to anything at all—for example an arbitrary location in the middle of a block of code, or a completely nonexistent virtual address.

For that reason, CCL:SAVE-APPLICATION converts all MACPTR objects to DEAD-MACPTR objects when writing them to an image file. A DEAD-MACPTR is functionally identical to a MACPTR, except that code that operates on MACPTR objects distinguishes them from DEAD-MACPTR objects and can handle them appropriately—signaling errors, for example.

As of Clozure CL 1.2, there is one exception to the conversion of MACPTR to DEAD-MACPTR objects: a MACPTR object that points to the address 0 is not converted, because address 0 can always be relied upon to refer to the same thing.

As of Clozure CL 1.2, the constant CCL:+NULL-PTR+ refers to a MACPTR object that points to address 0.

On all supported platforms, you can use CCL:SAVE-APPLICATION to create a command-line tool that runs the same way any command-line program does. Alternatively, if you choose not to prepend the kernel, you can save an image and then later run it by passing it as a command-line parameter to the ccl or ccl64 script.

SAVE-APPLICATION filename &key toplevel-function init-file error-handler application-class clear-clos-caches (purify t) impurify (mode #o644) prepend-kernel native [Function]

filename

The pathname of the file to be created when Clozure CL saves the application.

toplevel-function

The function to be executed after startup is complete. The toplevel is a function of no arguments that performs whatever actions the lisp system should perform when launched with this image.

If this parameter is not supplied, Clozure CL uses its default toplevel. The default toplevel runs the read-eval-print loop.

init-file

The pathname of a Lisp file to be loaded when the image starts up. You can place initialization expressions in this file, and use it to customize the behavior of the Lisp system when it starts up.

error-handler

The error-handling mode for the saved image. The supplied value determines what happens when an error is not handled by the saved image. Valid values are :quit (Lisp exits with an error message); :quit-quietly (Lisp exits without an error message); or :listener (Lisp enters a break loop, enabling you to debug the problem by interacting in a listener). If you don't supply this parameter, the saved image uses the default error handler (:listener).

application-class

The CLOS class that represents the saved Lisp application. Normally you don't need to supply this parameter; CCL:SAVE-APPLICATION uses the class CCL:LISP-DEVELOPMENT-SYSTEM. In some cases you may choose to create a custom application class; in that case, pass the name of the class as the value for this parameter.

clear-clos-caches

If true, ensures that CLOS caches are emptied before saving the image. Normally you don't need to supply this parameter, but if for some reason you want to ensure the CLOS caches are clear when the image starts up, you can pass any true value.

purify

When true, calls (in effect) purify before saving the heap image. This moves certain objects that are unlikely to become garbage to a special memory area that is not scanned by the GC (since it is expected that the GC wouldn't find anything to collect).

impurify

If true, calls (in effect) impurify before saving the heap image. (If both :impurify and :purify are true, first impurify is done, and then purify.)

impurify moves objects in certain special memory areas into the regular dynamic heap, where they will be scanned by the GC.

mode

A number specifying the mode (permission bits) of the output file.

prepend-kernel

Specifies the file to prepend to the saved heap image. A value of t means to prepend the lisp kernel binary that the lisp started with. Otherwise, the value of :prepend-kernel should be a pathname designator for the file to be prepended.

If the prepended file is execuatable, its execute mode bits will be copied to the output file.

This argument can be used to prepend any kind of file to the saved heap image. This can be useful in some special cases.

native

If true, saves the image as a native (ELF, Mach-O, PE) shared library. (On platforms where this isn't yet supported, a warning is issued and the option is ignored.)

4.10. Concatenating FASL Files

Multiple fasl files can be concatenated into a single file.

[Function]

fasl-concatenate out-file fasl-files &key (:if-exists :error)
Concatenate several fasl files, producing a single output file.

Arguments and Values:

out-file--- Name of the file in which to store the concatenation.

fasl-files--- List of names of fasl files to concatenate.

:if-exists--- As for OPEN, defaults to :error

Description:

Creates a fasl file which, when loaded, will have the same effect as loading the individual input fasl files in the specified order. The single file might be easier to distribute or install, and loading it may be at least a little faster than loading the individual files (since it avoids the overhead of opening and closing each file in succession.)

The PATHNAME-TYPE of the output file and of each input file defaults to the current platform's fasl file type (.dx64fsl or whatever.) If any of the input files has a different type/extension an error will be signaled, but it doesn't otherwise try too hard to verify that the input files are real fasl files for the current platform.

4.11. Floating Point Numbers

In Clozure CL, the Common Lisp types short-float and single-float are implemented as IEEE single precision values; double-float and long-float are IEEE double precision values. On 64-bit platforms, single-floats are immediate values (like fixnums and characters).

Floating-point exceptions are generally enabled and detected. By default, threads start up with overflow, division-by-zero, and invalid enabled, and the rounding mode is set to nearest. The functions SET-FPU-MODE and GET-FPU-MODE provide user control over floating-point behavior.

[Function]

get-fpu-mode &optional mode
Return the state of exception-enable and rounding-mode control flags for the current thread.

Arguments and Values:

mode--- One of the keywords :rounding-mode, :overflow, :underflow, :division-by-zero, :invalid, :inexact.

Description:

If mode is supplied, returns the value of the corresponding control flag for the current thread.

Otherwise, returns a list of keyword/value pairs which describe the floating-point exception-enable and rounding-mode control flags for the current thread.

rounding-mode--- One of :nearest, :zero, :positive, :negative

overflow, underflow, division-by-zero, invalid, inexact --- If true, the floating-point exception is signaled. If NIL, it is masked.

[Function]

set-fpu-mode &key rounding-mode overflow underflow division-by-zero invalid inexact
Set the state of exception-enable and rounding-mode control flags for the current thread.

Arguments and Values:

rounding-mode--- If supplied, must be one of :nearest, :zero, :positive, or :negative.

overflow, underflow, division-by-zero, invalid, inexact---NIL to mask the exception, T to signal it.

Description:

Sets the current thread's exception-enable and rounding-mode control flags to the indicated values for arguments that are supplied, and preserves the values assoicated with those that aren't supplied.

4.12. Watched Objects

As of release 1.4, Clozure CL provides a way for lisp objects to be watched so that a condition will be signaled when a thread attempts to write to the watched object. For a certain class of bugs (someone is changing this value, but I don't know who), this can be extremely helpful.

4.12.1. WATCH

[Function]

watch &optional object
Monitor a lisp object for writes.

Arguments and Values:

object--- Any memory-allocated lisp object.

Description:

The WATCH function arranges for the specified object to be monitored for writes. This is accomplished by copying the object to its own set of virtual memory pages, which are then write-protected. This protection is enforced by the computer's memory-management hardware; the write-protection does not slow down reads at all.

When any write to the object is attempted, a WRITE-TO-WATCHED-OBJECT condition will be signaled.

When called with no arguments, WATCH returns a freshly-consed list of the objects currently being watched.

WATCH returns NIL if the object cannot be watched (typically because the object is in a static or pure memory area).

DWIM:

WATCH operates at a fairly low level; it is not possible to avoid the details of the internal representation of objects. Nevertheless, as a convenience, WATCHing a standard-instance, a hash-table, or a multi-dimensional or non-simple CL array will watch the underlying slot-vector, hash-table-vector, or data-vector, respectively.

Discussion:

WATCH can monitor any memory-allocated lisp object.

In Clozure CL, a memory-allocated object is either a cons cell or a uvector.

WATCH operates on cons cells, not lists. In order to watch a chain of cons cells, each cons cell must be watched individually. Because each watched cons cell takes up its own own virtual memory page (4 Kbytes), it's only feasible to watch relatively short lists.

If a memory-allocated object isn't a cons cell, then it is a vector-like object called a uvector. A uvector is a memory-allocated lisp object whose first word is a header that describes the object's type and the number of elements that it contains.

So, a hash table is a uvector, as is a string, a standard instance, a double-float, a CL array or vector, and so forth.

Some CL objects, like strings and other simple vectors, map in a straightforward way onto the uvector representation. It is easy to understand what happens in such cases. The uvector index corresponds directly to the vector index:


? (defvar *s* "xxxxx")
*S*
? (watch *s*)
"xxxxx"
? (setf (char *s* 3) #\o)
> Error: Write to watched uvector "xxxxx" at index 3
>        Faulting instruction: (movl (% eax) (@ -5 (% r15) (% rcx)))
> While executing: SET-CHAR, in process listener(1).
> Type :POP to abort, :R for a list of available restarts.
> Type :? for other options.

    

In the case of more complicated objects (e.g., a hash-table, a standard-instance, a package, etc.), the elements of the uvector are like slots in a structure. It's necessary to know which one of those "slots" contains the data that will be changed when the object is written to.

As mentioned above, watch knows about arrays, hash-tables, and standard-instances, and will automatically watch the appropriate data-containing element.

An example might make this clearer.


? (defclass foo ()
    (slot-a slot-b slot-c))
#<STANDARD-CLASS FOO>
? (defvar *a-foo* (make-instance 'foo))
*A-FOO*
? (watch *a-foo*)
#<SLOT-VECTOR #xDB00D>
;;; Note that WATCH has watched the internal slot-vector object
? (setf (slot-value *a-foo* 'slot-a) 'foo)
> Error: Write to watched uvector #<SLOT-VECTOR #xDB00D> at index 1
>        Faulting instruction: (movq (% rsi) (@ -5 (% r8) (% rdi)))
> While executing: %MAYBE-STD-SETF-SLOT-VALUE-USING-CLASS, in process listener(1).
> Type :POP to abort, :R for a list of available restarts.
> Type :? for other options.

    

Looking at a backtrace would presumably show what object and slot name were written.

Note that even though the write was to slot-a, the uvector index was 1 (not 0). This is because the first element of a slot-vector is a pointer to the instance that owns the slots. We can retrieve that to look at the object that was modified:


1 > (uvref (write-to-watched-object-object *break-condition*) 0)
#<FOO #x30004113502D>
1 > (describe *)
#<FOO #x30004113502D>
Class: #<STANDARD-CLASS FOO>
Wrapper: #<CLASS-WRAPPER FOO #x300041135EBD>
Instance slots
SLOT-A: #<Unbound>
SLOT-B: #<Unbound>
SLOT-C: #<Unbound>
1 >
 
    

4.12.2. UNWATCH

[Function]

unwatch object
Stop monitoring a lisp object for writes.

Description:

The UNWATCH function ensures that the specified object is in normal, non-monitored memory. If the object is not currently being watched, UNWATCH does nothing and returns NIL. Otherwise, the newly unwatched object is returned.

4.12.3. WRITE-TO-WATCHED-OBJECT

[Condition]

WRITE-TO-WATCHED-OBJECT
Condition signaled when a write to a watched object is attempted.

Discussion:

This condition is signaled when a watched object is written to. There are three slots of interest:

object--- The actual object that was the destination of the write.

offset--- The byte offset from the tagged object pointer to the address of the write.

instruction--- The disassembled machine instruction that attempted the write.

Restarts:

A few restarts are provided: one will skip over the faulting write instruction and proceed; another offers to unwatch the object and continue.

There is also an emulate restart. In some common cases, the faulting write instruction can be emulated, enabling the write to be performed without having to unwatch the object (and therefore let other threads potentially write to it). If the faulting instruction isn't recognized, the emulate restart will not be offered.

4.12.4. Notes

Although some care has been taken to minimize potential problems arising from watching and unwatching objects from multiple threads, there may well be subtle race conditions present that could cause bad behavior.

For example, suppose that a thread attempts to write to a watched object. This causes the operating system to generate an exception. The lisp kernel figures out what the exception is, and calls back into lisp to signal the write-to-watched-object condition and perhaps handle the error.

Now, as soon lisp code starts running again (for the callback), it's possible that some other thread could unwatch the very watched object that caused the exception, perhaps before we even have a chance to signal the condition, much less respond to it.

Having the object unwatched out from underneath a handler may at least confuse it, if not cause deeper trouble. Use caution with unwatch.

4.12.5. Examples

Here are a couple more examples in addition to the above examples of watching a string and a standard-instance.

4.12.5.1. Fancy arrays

?  (defvar *f* (make-array '(2 3) :element-type 'double-float))
*F*
? (watch *f*)
#(0.0D0 0.0D0 0.0D0 0.0D0 0.0D0 0.0D0)
;;; Note that the above vector is the underlying data-vector for the array
? (setf (aref *f* 1 2) pi)
> Error: Write to watched uvector #<VECTOR 6 type DOUBLE-FLOAT, simple> at index 5
>        Faulting instruction: (movq (% rax) (@ -5 (% r8) (% rdi)))
> While executing: ASET, in process listener(1).
> Type :POP to abort, :R for a list of available restarts.
> Type :? for other options.
1 > 
  

In this case, uvector index in the report is the row-major index of the element that was written to.

4.12.5.2. Hash tables

Hash tables are surprisingly complicated. The representation of a hash table includes an element called a hash-table-vector. The keys and values of the elements are stored pairwise in this vector.

One problem with trying to monitor hash tables for writes is that the underlying hash-table-vector is replaced with an entirely new one when the hash table is rehashed. A previously-watched hash-table-vector will not be the used by the hash table after rehashing, and writes to the new vector will not be caught.

? (defvar *h* (make-hash-table))
*H*
? (setf (gethash 'noise *h*) 'feep)
FEEP
? (watch *h*)
#<HASH-TABLE-VECTOR #xDD00D>
;;; underlying hash-table-vector
? (setf (gethash 'noise *h*) 'ding)
> Error: Write to watched uvector #<HASH-TABLE-VECTOR #xDD00D> at index 35
>        Faulting instruction: (lock)
>          (cmpxchgq (% rsi) (@ (% r8) (% rdx)))
> While executing: %STORE-NODE-CONDITIONAL, in process listener(1).
> Type :POP to abort, :R for a list of available restarts.
> Type :? for other options.
;;; see what value is being replaced...
1 > (uvref (write-to-watched-object-object *break-condition*) 35)
FEEP
;;; backtrace shows useful context
1 > :b
*(1A109F8) : 0 (%STORE-NODE-CONDITIONAL ???) NIL
 (1A10A50) : 1 (LOCK-FREE-PUTHASH NOISE #<HASH-TABLE :TEST EQL size 1/60 #x30004117D47D> DING) 653
 (1A10AC8) : 2 (CALL-CHECK-REGS PUTHASH NOISE #<HASH-TABLE :TEST EQL size 1/60 #x30004117D47D> DING) 229
 (1A10B00) : 3 (TOPLEVEL-EVAL (SETF (GETHASH # *H*) 'DING) NIL) 709
 ...
  

4.12.5.3. Lists

As previously mentioned, WATCH only watches individual cons cells.

? (defun watch-list (list)
    (maplist #'watch list))
WATCH-LIST
? (defvar *l* (list 1 2 3))
*L*
? (watch-list *l*)
((1 2 3) (2 3) (3))
? (setf (nth 2 *l*) 'foo)
> Error: Write to the CAR of watched cons cell (3)
>        Faulting instruction: (movq (% rsi) (@ 5 (% rdi)))
> While executing: %SETNTH, in process listener(1).
> Type :POP to abort, :R for a list of available restarts.
> Type :? for other options.
  

4.13. Code Coverage

4.13.1. Overview

In Clozure CL 1.4 and later, code coverage provides information about which paths through generated code have been executed and which haven't. For each source form, it can report one of three possible outcomes:

  • Not covered: this form was never entered.

  • Partly covered: This form was entered, and some parts were executed and some weren't.

  • Fully covered: Every bit of code generated from this form was executed.

4.13.2. Limitations

While the information gathered for coverage of generated code is complete and precise, the mapping back to source forms is of necessity heuristic, and depends a great deal on the behavior of macros and the path of the source forms through compiler transforms. Source information is not recorded for variables, which further limits the source mapping. In practice, there is often enough information scattered about a partially covered function to figure out which logical path through the code was taken and which wasn't. If that doesn't work, you can try disassembling to see which parts of the compiled code were not executed: in the disassembled code there will be references to #<CODE-NOTE [xxx] ...> where xxx is NIL if the code that follows was never executed and non-NIL if it was.

Sometimes the situation can be improved by modifying macros to try to preserve more of the input forms, rather than destructuring and rebuilding them.

Because the code coverage information is associated with compiled functions, code coverage information is not available for load-time toplevel expressions. You can work around this by creating a function and calling it. I.e. instead of

(progn
  (do-this)
  (setq that ...) ...))

do:

(defun init-this-and-that ()
  (do-this)
  (setq that ...)  ...)
(init-this-and-that)

Then you can see the coverage information in the definition of init-this-and-that.

4.13.3. Usage

In order to gather code coverage information, you first have to recompile all your code to include code coverage instrumentation. Compiling files will generate code coverage instrumentation if CCL:*COMPILE-CODE-COVERAGE* is true:

(setq ccl:*compile-code-coverage* t) 
(recompile-all-your-files)

The compilation process will be many times slower than normal, and the fasl files will be many times bigger.

When you execute functions loaded from instrumented fasl files, they will record coverage information every time they are executed. You can examine that information by calling ccl:report-coverage or ccl:coverage-statistics.

While recording coverage, you can collect incremental coverage deltas between any two points in time. You might do this while running a test suite, to record the coverage for each test, for example:

(ccl:reset-incremental-coverage)
(loop with coverage = (make-hash-table)
      for test in (tests-to-run)
      do (run-test test)
      do (setf (gethash test coverage) (ccl:get-incremental-coverage))
      finally (return coverage))

creates a hash table mapping a test to a representation of all coverage recorded while running the test. This hash table can then be passed to ccl:report-coverage, ccl:incremental-coverage-svn-matches or ccl:incremental-coverage-source-matches.

4.13.4. Functions and Variables

The following functions can be used to manage the coverage data:

[Function]

report-coverage output-file &key (tags nil) (external-format :default) (statistics t) (html t)
Generate a code coverage report

Arguments and Values:

output-file--- Pathname for the output index file.

html--- If non-nil (the default), this will generate an HTML report, consisting of an index file in output-file and, in the same directory, one html file for each instrumented source file that has been loaded in the current session.

tags--- If non-nil, this should be a hash table mapping arbitrary keys (tags) to incremental coverage deltas. The HTML report will show a list of tags, and allow selection of an arbitrary subset of them to show the coloring and statistics for coverage by that subset.

external-format--- Controls the external format of the html files.

statistics--- If non-nil (the default), a comma-separated file is generated with the summary of statistics. You can specify a filename for the statistics argument, otherwise "statistics.csv" is created in the directory of output-file. See documentation of coverage-statistics below for a description of the values in the statistics file.

Example:

If you've loaded foo.lx64fsl and bar.lx64fsl, and have run some tests, you could do

(REPORT-COVERAGE "/my/dir/coverage/report.html")
    

and this would generate report.html, foo_lisp.html and bar_lisp.html, and statistics.csv all in /my/dir/coverage/.

[Function]

RESET-COVERAGE
Resets all coverage data back to the "Not Executed" state

Description:

Resets all coverage data back to the "Not Executed" state

[Function]

CLEAR-COVERAGE
Forget about all instrumented files that have been loaded.

Description:

Gets rid of the information about which instrumented files have been loaded, so ccl:report-coverage will not report any files, and ccl:save-coverage-in-file will not save any info, until more instrumented files are loaded.

[Function]

save-coverage-in-file pathname
Save all coverage into to a file so you can restore it later.

Description:

Saves all coverage info in a file, so you can restore the coverage state later. This allows you to combine multiple runs or continue in a later session. Equivalent to (ccl:write-coverage-to-file (ccl:get-coverage) pathname).

[Function]

restore-coverage-from-file pathname
Load coverage state from a file.

Description:

Restores the coverage data previously saved with ccl:save-coverage-in-file, for the set of instrumented fasls that were loaded both at save and restore time. I.e. coverage info is only restored for files that have been loaded in this session. For example if in a previous session you had loaded "foo.lx86fsl" and then saved the coverage info, in this session you must load the same "foo.lx86fsl" before calling restore-coverage-from-file in order to retrieve the stored coverage info for "foo". Equivalent to (ccl:restore-coverage (ccl:read-coverage-from-file pathname)).

[Function]

GET-COVERAGE
Returns a snapshot of the current coverage data.

Description:

Returns a snapshot of the current coverage data. A snapshot is a copy of the current coverage state. It can be saved in a file with ccl:write-coverage-to-file, reinstated back as the current state with ccl:restore-coverage, or combined with other snapshots with ccl:combine-coverage.

[Function]

restore-coverage snapshot
Reinstalls a coverage snapshot as the current coverage state.

Description:

Reinstalls a coverage snapshot as the current coverage state.

[Function]

combine-coverage snapshots
Combines multiple coverage snapshots into one.

Description:

Takes a list of coverage snapshots and returns a new coverage snapshot representing a union of all the coverage data.

[Function]

write-coverage-to-file snapshot pathname
Save a coverage snapshot in a file.

Description:

Saves the coverage snapshot in a file. The snapshot can be loaded back with ccl:read-coverage-from-file or loaded and restored with ccl:restore-coverage-from-file. Note that the file created is actually a lisp source file and can be compiled for faster loading.

[Function]

read-coverage-from-file pathname
Return the coverage snapshot saved in a file.

Description:

Returns the snapshot saved in pathname. Doesn't affect the current coverage state. pathname can be the file previously created with ccl:write-coverage-to-file or ccl:save-coverage-in-file, or it can be the name of the fasl created from compiling such a file.

[Function]

coverage-statistics
Returns a sequence of ccl:coverage-statistics objects, one per source file.

Description:

Returns a sequence of ccl:coverage-statistics objects, one for each source file, containing the same information as that written to the statistics file by ccl:report-coverage. The following accessors are defined for ccl:coverage-statistics objects:

coverage-source-file

the name of the source file corresponding to this information

coverage-expressions-total

the total number of expressions

coverage-expressions-entered

the number of source expressions that have been entered (i.e. at least partially covered)

coverage-expressions-covered

the number of source expressions that were fully covered

coverage-unreached-branches

the number of conditionals with one branch taken and one not taken

coverage-code-forms-total

the total number of code forms. A code form is an expression in the final stage of compilation, after all macroexpansion and compiler transforms and simplification

coverage-code-forms-covered

the number of code forms that have been entered

coverage-functions-total

the total number of functions

coverage-functions-fully-covered

the number of functions that were fully covered

coverage-functions-partly-covered

the number of functions that were partly covered

coverage-functions-not-entered

the number of functions never entered

[Function]

reset-incremental-coverage
Reset incremental coverage.

Description:

Marks a starting point for recording incremental coverage. Note that calling this function does not affect regular coverage data (whereas calling ccl:reset-coverage resets incremental coverage as well).

[Function]

get-incremental-coverage &key (reset t)
Returns the delta of coverage since the last incremental reset.

Description:

Returns the delta of coverage since the last reset of incremental coverage. If reset is true (the default), it also resets incremental coverage now, so that the next call to get-incremental-coverage will return the delta from this point.

Incremental coverage deltas are represented differently than the full coverage snapshots returned by functions such as ccl:get-coverage. Incremental coverage uses an abbreviated format and is missing some of the information in a full snapshot, and therefore cannot be passed to functions documented to accept a snapshot, only to functions specifically documented to accept incremental coverage deltas.

[Function]

incremental-coverage-source-matches collection sources
Find incremental coverage deltas intersecting source regions.

Arguments and Values:

collection--- A hash table mapping arbitrary keys to incremental coverage deltas, or a sequence of incremental coverage deltas.

sources--- A list of pathnames and/or source-notes, the latter representing a range within a file.

Description:

Given a hash table collection whose values are incremental coverage deltas, return a list of all keys corresponding to those deltas that intersect any region in sources.

For example if the deltas represent tests, then the returned value is a list of all tests that cover some part of the source regions.

collection can also be a sequence of deltas, in which case a subsequence of matching deltas is returned. In particular you can test whether any particular delta intersects the sources by passing it in as a single-element list.

[Function]

incremental-coverage-svn-matches collection &key (directory (current-directory)) (revision :base)
Find incremental coverage deltas matching changes from a particular subversion revision.

Arguments and Values:

collection--- A hash table mapping arbitrary keys to incremental coverage deltas, or a sequence of incremental coverage deltas.

directory--- The pathname of a subversion working directory.

revision--- The revision to compare to the working directory, an integer or another value whose printed representation is suitable for passing as the --revision argument to svn.

Description:

Given a hash table collection whose values are incremental coverage deltas, return a list of all keys corresponding to those deltas that intersect any changed source in directory since revision revision in subversion.

For example if the deltas represent tests, then the returned value is a list of all tests that might be affected by the changes.

collection can also be a sequence of deltas, in which case a subsequence of matching deltas is returned. In particular you can test whether any particular delta is affected by the changes by passing it in as a single-element list.

[Variable]

*compile-code-coverage*
When true, instrument functions being compiled to collect code coverage information.

Description:

This variable controls whether functions are instrumented for code coverage. Files compiled while this variable is true will contain code coverage instrumentation.

[Macro]

without-compiling-code-coverage
Don't record code coverage for forms within the body.

Description:

This macro arranges so that body doesn't record internal details of code coverage. It will be considered totally covered if it's entered at all. The Common Lisp macros ASSERT and CHECK-TYPE use this macro.

4.13.5. Interpreting Code Coloring

The output of ccl:report-coverage consists of formatted source code, with coverage indicated by coloring. Four colors are used: dark green for forms that compiled to code in which every single instruction was executed, light green for forms that have been entered but weren't totally covered, red for forms that were never entered, and the page background color for toplevel forms that weren't instrumented.

The source coloring is applied from outside in. So for example if you have

(outer-form ... (inner-form ...) ...)
  

first the whole outer form is painted with whatever color expresses the outer form coverage, and then the inner form color is replaced with whatever color expresses the inner form coverage. One consequence of this approach is that every part of the outer form that is not specifically inside some executable inner form will have the outer form's coverage color. If the syntax of outer form involves some non-executable forms, or forms that do not have coverage info of their own for whatever reason, then they will just inherit the color of the outer form, because they don't get repainted with a color of their own.

One case in which this approach can be confusing is in the case of symbols. As noted in the Limitations section, coverage information is not recorded for variables; hence the coloring of a variable does not convey information about whether the variable was evaluated or not -- that information is not available, and the variable just inherits the color of the form that contains it.

4.14. Other Extensions

[Function]

quit &optional (exit 0) &key error-handler
Cleanly exit the lisp.

Summary:

Cleanly exit from lisp. If the exit argument is a value of type (signed-byte 32), that value will be passed to the C library function _exit() as the status code. (A value of nil is treated as a zero.)

Alternatively, exit may be a function of no arguments; this function will be called instead of _exit() to exit the lisp.

The error-handler argument, if supplied, must be a function of one argument, the condition, that will be called if an error occurs when preparing to quit. The error-handler function should exit the lisp.

[Function]

wait-for-signal s duration
Wait for a particular signal to be received.

Summary:

Wait for the signal with signal number s to be received, or until duration seconds have elapsed. If duration is nil, wait for an indeterminate "very long time" (many years).

If signal number s is outside the range of valid signals, or is reserved by the lisp for its own use, an error is signaled. (An error is always signaled on Windows systems.)

[Special operator]

unwind-protect protected-form {cleanup-form}*
Ensure cleanup-forms are executed.

Summary:

In Clozure CL, the cleanup forms are always executed as if they were wrapped with without-interrupts. To allow interrupts, use with-interrupts-enabled.

[Variable]

*disassemble-verbose*
When true, display verbose disassembly output.

Description:

When true, the output of disassemble may include platform-dependent additional information. For instance, on the x86 ports, the output will include the x86 opcode bytes.

The default value is nil.

Chapter 5. The Clozure CL IDE

5.1. Introduction

Clozure CL ships with the complete source code for an integrated development environment written using Cocoa on Mac OS X. This chapter describes how to build and use that environment, referred to hereafter simply as "the IDE".

The IDE provides a programmable text editor, listener windows, an inspector for Lisp data structures, and a means of easily building a Cocoa application in Lisp. In addition, its source code provides an example of a fairly complex Cocoa application written in Lisp.

The current version of the IDE has seen the addition of numerous features and many bugfixes. Although it's by no means a finished product, we hope it will prove more useful than previous versions, and we plan additional work on the IDE for future releases.

5.2. Building the IDE

Building the Clozure CL IDE is now a very simple process.

  1. In a shell session, cd to the ccl directory.

  2. Run ccl from the shell. The easiest way to do this is generally to execute the ccl or ccl64 command.

  3. Evaluate the form (require :cocoa-application)

For example, assuming that the Clozure CL distribution is installed in "/usr/local/ccl", the following sequence of shell interactions builds the IDE:

oshirion:ccl mikel$ ccl64
Welcome to Clozure Common Lisp Version 1.2-r9198M-trunk  (DarwinX8664)!
? (require :cocoa-application)
;Loading #P"ccl:cocoa-ide;fasls;cocoa-utils.dx64fsl.newest"...
;Loading #P"ccl:cocoa-ide;fasls;cocoa-defaults.dx64fsl.newest"...

[...many lines of "Compiling" and "Loading" omitted...]

Saving application to /usr/local/ccl/Clozure CL.app/

oshirion:ccl mikel$ 

    

Clozure CL compiles and loads the various subsystems that make up the IDE, then constructs a Cocoa application bundle named "Clozure CL.app" and saves the Lisp image into it. Normally Clozure CL creates the application bundle in the root directory of the Clozure CL distribution.

5.3. Running the IDE

After it has been built, you can run the "Clozure CL.app" application normally, by double-clicking its icon. When launched, the IDE initially displays a single listener window that you can use to interact with Lisp. You can type Lisp expressions for evaluation at the prompt in the listener window. You can also use Hemlock editing commands to edit the text of expressions in the listener window.

5.4. IDE Features

5.4.1. Editor Windows

You can open an editor window either by choosing Open from the File menu and then selecting a text file, or by choosing New from the File menu. You can also evaluate the expression (ed) in the listener window; in that case Clozure CL creates a new window as if you had chosen New from the File menu.

Editor windows implement Hemlock editing commands. You can use all the editing and customization features of Hemlock within any editor window (including listener windows).

5.4.2. The Lisp Menu

The Lisp menu provides several commands for interacting with the running Lisp session, in addition to the ways you can interact with it by evaluating expressions. You can evaluate a selected range of text in any editing buffer. You can compile and load the contents of editor windows (please note that in the current version, Clozure CL compiles and loads the contents of the file associated with an editor window; that means that if you try to load or compile a window that has not been saved to a file, the result is an error).

You can interrupt computations, trigger breaks, and select restarts from the Lisp menu. You can also display a backtrace or open the Inspector window.

5.4.2.1. Checking for Updates

At the bottom of the Lisp menu is an item entitled "Check for Updates". If your copy of Clozure CL came from the Clozure Subversion server (which is the preferred source), and if your internet connection is working, then you can select this menu item to check for updates to your copy of Clozure CL.

When you select "Check for Updates", Clozure CL uses the svn program to query the Clozure Subversion repository and determine whether new updates to Clozure CL are available. (This means that on Mac OS X versions earlier than 10.5, you must ensure that the Subversion client software is installed before using the "Check for Updates" feature. See the wikiHow page on installing Subversion for more information.) If updates are available, Clozure CL automatically downloads and installs them. After a successful download, Clozure CL rebuilds itself, and then rebuilds the IDE on the newly-rebuilt Lisp. Once this process is finished, you should quit the running IDE and start the newly built one (which will be in the same place that the old one was).

Normally, Clozure CL can install updates and rebuild itself without any problems. Occasionally, an unforeseen problem (such as a network outage, or a hardware failure) might interrupt the self-rebuilding process, and leave your copy of Clozure CL unusable. If you are expecting to update your copy of Clozure CL frequently, it might be prudent to keep a backup copy of your working environment ready in case of such situtations. You can also always obtain a full, fresh copy of Clozure CL from Clozure's repository..

5.4.3. The Tools Menu

The tools menu provides access to the Apropos and Processes windows. The Apropos window searches the running Lisp image for symbols that match any text you enter. You can use the Apropos window to quickly find function names and other useful symbols. The Processes window lists all threads running in the current Lisp session. If you double-click a process entry, Clozure CL opens an Inspector window on that process.

5.4.4. The Inspector Window

The Inspector window displays information about a Lisp value. The information displayed varies from the very simple, in the case of a simple data value such as a character, to the complex, in the case of structured data such as lists or CLOS objects. The left-hand column of the window's display shows the names of the object's attributes; the righthand column shows the values associated with those attributes. You can inspect the values in the righthand column by double-clicking them.

Inspecting a value in the righthand column changes the Inspector window to display the double-clicked object. You can quickly navigate the fields of structured data this way, inspecting objects and the objects that they refer to. Navigation buttons at the top left of the window enable you to retrace your steps, backing up to return to previously-viewed objects, and going forward again to objects you navigated into previously.

You can change the contents of a structured object by evaluating expressions in a listener window. The refresh button (marked with a curved arrow) updates the display of the Inspector window, enabling you to quickly see the results of changing a data structure.

5.5. IDE Sources

Clozure CL builds the IDE from sources in the "objc-bridge" and "cocoa-ide" directories in the Clozure CL distribution. The IDE as a whole is a relatively complicated application, and is probably not the best place to look when you are first trying to understand how to build Cocoa applications. For that, you might benefit more from the examples in the "examples/cocoa/" directory. Once you are familiar with those examples, though, and have some experience building your own application features using Cocoa and the Objective-C bridge, you might browse through the IDE sources to see how it implements its features.

The search path for Clozure CL's REQUIRE feature includes the "objc-bridge" and "cocoa-ide" directories. You can load features defined in these directories by using REQUIRE. For example, if you want to use the Cocoa features of Clozure CL from a terminal session (or from an Emacs session using SLIME or ILISP), you can evaluate (require :cocoa).

5.6. The Application Builder

One important feature of the IDE currently has no Cocoa user interface: the application builder. The application builder constructs a Cocoa application bundle that runs a Lisp image when double-clicked. You can use the application builder to create Cocoa applications in Lisp. These applications are exactly like Cocoa applications created with XCode and Objective-C, except that they are written in Lisp.

To make the application builder available, evaluate the expression (require :build-application). Clozure CL loads the required subsystems, if necessary.

BUILD-APPLICATION &key (name "MyApplication") (type-string "APPL") (creator-string "OMCL") (directory (current-directory)) (copy-ide-resources t) (info-plist NIL) (nibfiles NIL) (main-nib-name NIL) (application-class 'GUI::COCOA-APPLICATION) (toplevel-function NIL) [Function]

The build-application function constructs an application bundle, populates it with the files needed to satisfy Mac OS X that the bundle is a launchable application, and saves an executable Lisp image to the proper subdirectory of the bundle. Assuming that the saved Lisp image contains correct code, a user can subsequently launch the resulting Cocoa application by double-clicking its icon in the Finder, and the saved Lisp environment runs.

The keyword arguments control various aspects of application bundle as BUILD-APPLICATION builds it.

name

Specifies the application name of the bundle. BUILD-APPLICATION creates an application bundle whose name is given by this parameter, with the extension ".app" appended. For example, using the default value for this parameter results in a bundle named "MyApplication.app".

type-string

Specifies type of bundle to create. You should normally never need to change the default value, which Mac OS X uses to identify application bundles.

creator-string

Specifies the creator code, which uniquely identifies the application under Mac OS X. The default creator code is that of Clozure CL. For more information about reserving and assigning creator codes, see Apple's developer page on the topic.

directory

The directory in which BUILD-APPLICATION creates the application bundle. By default, it creates the bundle in the current working directory. Unless you use CURRENT-DIRECTORY to set the working directory, the bundle may be created in some unexpected place, so it's safest to specify a full pathname for this argument. A typical value might be "/Users/foo/Desktop/" (assuming, of course, that your username is "foo").

copy-ide-resources

Whether to copy the resource files from the IDE's application bundle. By default, BUILD-APPLICATION copies nibfiles and other resources from the IDE to the newly-created application bundle. This option is often useful when you are developing a new application, because it enables your built application to have a fully-functional user interface even before you have finished designing one. By default, the application uses the application menu and other UI elements of the IDE until you specify otherwise. Once your application's UI is fully implemented, you may choose to pass NIL for the value of this parameter, in which case the IDE resources are not copied into your application bundle.

info-plist

A user-supplied NSDictionary object that defines the contents of the Info.plist file to be written to the application bundle. The default value is NIL, which specifies that the Info.plist from the IDE is to be used if copy-ide-resources is true, and a new dictionary created with default values is to be used otherwise. You can create a suitable NSDictionary object using the function make-info-dict. For details on the parameters to this function, see its definition in "ccl/cocoa-ide/builder-utilities.lisp".

nibfiles

A list of pathnames, where each pathname identifies a nibfile created with Apple's InterfaceBuilder application. BUILD-APPLICATION copies each nibfile into the appropriate place in the application bundle, enabling the application to load user-interface elements from them as-needed. It is safest to provide full pathnames to the nibfiles in the list. Each nibfile must be in ".nib" format, not ".xib" format, in order that the application can load it.

main-nib-name

The name of the nibfile to load initially when launching. The user-interface defined in this nibfile becomes the application's main interface. You must supply the name of a suitable nibfile for this parameter, or the resulting application uses the Clozure CL user interface.

application-class

The name of the application's CLOS class. The default value is the class provided by Clozure CL for graphical applications. Supply the name of your application class if you implement one. If not, Clozure CL uses the default class.

toplevel-function

The toplevel function that runs when the application launches. Normally the default value, which is Clozure CL's toplevel, works well, but in some cases you may wish to customize the behavior of the application's toplevel. The best source of information about writing your own toplevel is the Clozure CL source code, especially the implementations of TOPLEVEL-FUNCTION in "ccl/level-1/l1-application.lisp"

BUILD-APPLICATION creates a folder named "name.app" in the directory directory. Inside that folder, it creates the "Contents" folder that Mac OS X application bundles are expected to contain, and populates it with the "MacOS" and "Resources" folders, and the "Info.plist" and "PkgInfo" files that must be present in a working application bundle. It takes the contents of the "Info.plist" and "PkgInfo" files from the parameters to BUILD-APPLICATION. If copy-ide-resources is true then it copies the contents of the "Resources" folder from the "Resources" folder of the running IDE.

The work needed to produce a running Cocoa application is very minimal. In fact, if you supply BUILD-APPLICATION with a valid nibfile and pathnames, it builds a running Cocoa application that displays your UI. It doesn't need you to write any code at all to do this. Of course, the resulting application doesn't do anything apart from displaying the UI defined in the nibfile. If you want your UI to accomplish anything, you need to write the code to handle its events. But the path to a running application with your UI in it is very short indeed.

Please note that BUILD-APPLICATION is a work in progress. It can easily build a working Cocoa application, but it still has limitations that may in some cases prove inconvenient. For example, in the current version it provides no easy way to specify an application delegate different from the default. If you find the current limitations of BUILD-APPLICATION too restrictive, and want to try extending it for your use, you can find the source code for it in "ccl/cocoa-ide/build-application.lisp". You can see the default values used to populate the "Info.plist" file in "ccl/cocoa-ide/builder-utilities.lisp".

For more information on how to use BUILD-APPLICATION, see the Currency Converter example in "ccl/examples/cocoa/currency-converter/".

5.6.1. Running the Application Builder From the Command Line

It's possible to automate use of the application builder by running a call to CCL:BUILD-APPLICATION from the terminal command line. For example, the following command, entered at a shell prompt in Mac OS X's Terminal window, builds a working copy of the Clozure CL environment called "Foo.app":

ccl -b -e "(require :cocoa)" -e "(require :build-application)" -e "(ccl::build-application :name \"Foo\")"
      

You can use the same method to automate building your Lisp/Cocoa applications. Clozure CL handles each Lisp expressions passed with a -e argument in order, so you can simply evaluate a sequence of Lisp expressions as in the above example to build your application, ending with a call to CCL:BUILD-APPLICATION. The call to CCL:BUILD-APPLICATION can process all the same arguments as if you evaluated it in a Listener window in the Clozure CL IDE.

Building a substantial Cocoa application (rather than just reproducing the Lisp environment using defaults, as is done in the above example) is likely to involve a relatively complicated sequence of loading source files and perhaps evaluating Lisp forms. You might be best served to place your command line in a shell script that you can more easily edit and test.

One potentially complicated issue concerns loading all your Lisp source files in the right order. You might consider using ASDF to define and load a system that includes all the parts of your application before calling CCL:BUILD-APPLICATION. ASDF is a "another system-definition facility", a sort of make for Lisp, and is included in the Clozure CL distribution. You can read more about ASDF at the ASDF home page.

Alternatively, you could use the standard features of Common Lisp to load your application's files in the proper order.

Chapter 6. Hemlock Programming

6.1. Introduction
6.2. Representation of Text
6.2.1. Lines
6.2.2. Marks
6.2.3. Regions
6.3. Buffers
6.3.1. The Current Buffer
6.3.2. Buffer Functions
6.3.3. Modelines
6.4. Altering and Searching Text
6.4.1. Altering Text
6.4.2. Text Predicates
6.4.3. Kill Ring
6.4.4. Active Regions
6.4.5. Searching and Replacing
6.5. The Current Environment
6.5.1. Different Scopes
6.5.2. Shadowing
6.6. Hemlock Variables
6.6.1. Variable Names
6.6.2. Variable Functions
6.6.3. Hooks
6.7. Commands
6.7.1. Introduction
6.7.2. The Command Interpreter
6.7.3. Command Types
6.7.4. Command Arguments
6.8. Modes
6.8.1. Mode Hooks
6.8.2. Major and Minor Modes
6.8.3. Mode Functions
6.9. Character Attributes
6.9.1. Introduction
6.9.2. Character Attribute Names
6.9.3. Character Attribute Functions
6.9.4. Character Attribute Hooks
6.9.5. System Defined Character Attributes
6.10. Controlling the Display
6.10.1. Views
6.10.2. The Current View
6.10.3. View Functions
6.10.4. Cursor Positions
6.10.5. Redisplay
6.11. Logical Key-Events
6.11.1. Introduction
6.11.2. Logical Key-Event Functions
6.11.3. System Defined Logical Key-Events
6.12. The Echo Area
6.12.1. Echo Area Functions
6.12.2. Prompting Functions
6.12.3. Control of Parsing Behavior
6.12.4. Defining New Prompting Functions
6.12.5. Some Echo Area Commands
6.13. Files
6.13.1. File Options and Type Hooks
6.13.2. Pathnames and Buffers
6.13.3. File Groups
6.13.4. File Reading and Writing
6.14. Hemlock's Lisp Environment
6.14.1. Entering and Leaving the Editor
6.14.2. Keyboard Input
6.14.3. Hemlock Streams
6.14.4. Interface to the Error System
6.14.5. Definition Editing
6.14.6. Event Scheduling
6.14.7. Miscellaneous
6.15. High-Level Text Primitives
6.15.1. Indenting Text
6.15.2. Lisp Text Buffers
6.15.3. English Text Buffers
6.15.4. Logical Pages
6.15.5. Filling
6.16. Utilities
6.16.1. String-table Functions
6.16.2. Ring Functions
6.16.3. Undoing commands
6.17. Miscellaneous
6.17.1. Key-events

6.1. Introduction

Hemlock is the text editor used in Clozure CL. It was originally based on the CMU Hemlock editor, but has since diverged from it in various ways. We continue to call the editor part of our IDE Hemlock to give credit where credit is due, but we make no attempt at source or API compatibility with the original Hemlock.

Like the code, this documentation is based on the original Hemlock documentation, modified as necessary.

Hemlock follows in the tradition of Emacs-compatible editors, with a rich set of extensible commands. This document describes the API for implementing new commands. The basic editor consists of a set of Lisp utility functions for manipulating buffers and the other data structures of the editor. All user level commands are written in terms of these functions. To find out how to define commands see Commands.

6.2. Representation of Text

In Hemlock, text is represented as a sequence of lines. Newline characters are never stored but are implicit between lines. The implicit newline character is treated as the single character #\Newline by the text primitives.

Text is broken into lines when it is first introduced into Hemlock. Text enters Hemlock from the outside world in two ways: reading a file, or pasting text from the system clipboard. Hemlock uses heuristics (which should be documented here!) to decide what newline convention to use to convert the incoming text into its internal representation as a sequence of lines. Similarly it uses heuristics (which should be documented here!) to convert the internal representation into a string with embedded newlines in order to write a file or paste a region into the clipboard.

6.2.1. Lines

A line is an object representing a sequence of characters with no line breaks.

[Function]

linep line

Description:

This function returns t if line is a line object, otherwise nil.

[Function]

line-string line

Description:

Given a line, this function returns as a simple string the characters in the line. This is setf'able to set the line-string to any string that does not contain newline characters. It is an error to destructively modify the result of line-string or to destructively modify any string after the line-string of some line has been set to that string.

[Function]

line-previous line

Description:

[Function]

line-next line

Description:

Given a line, line-previous returns the previous line or nil if there is no previous line. Similarly, line-next returns the line following line or nil.

[Function]

line-buffer line

Description:

This function returns the buffer which contains this line. Since a line may not be associated with any buffer, in which case line-buffer returns nil.

[Function]

line-length line

Description:

This function returns the number of characters in the line. This excludes the newline character at the end.

[Function]

line-character line index

Description:

This function returns the character at position index within line. It is an error for index to be greater than the length of the line or less than zero. If index is equal to the length of the line, this returns a #\newline character.

[Function]

line-plist line

Description:

This function returns the property-list for line. setf, getf, putf and remf can be used to change properties. This is typically used in conjunction with line-signature to cache information about the line's contents.

[Function]

line-signature line

Description:

This function returns an object that serves as a signature for a line's contents. It is guaranteed that any modification of text on the line will result in the signature changing so that it is not eql to any previous value. The signature may change even when the text remains unmodified, but this does not happen often.

6.2.2. Marks

A mark indicates a specific position within the text represented by a line and a character position within that line. Although a mark is sometimes loosely referred to as pointing to some character, it in fact points between characters. If the charpos is zero, the previous character is the newline character separating the previous line from the mark's line. If the charpos is equal to the number of characters in the line, the next character is the newline character separating the current line from the next. If the mark's line has no previous line, a mark with charpos of zero has no previous character; if the mark's line has no next line, a mark with charpos equal to the length of the line has no next character.

This section discusses the very basic operations involving marks, but a lot of Hemlock programming is built on altering some text at a mark. For more extended uses of marks see Altering And Searching Text.

6.2.2.1. Kinds of Marks

A mark may have one of two lifetimes: temporary or permanent. Permanent marks remain valid after arbitrary operations on the text; temporary marks do not. Temporary marks are used because less bookkeeping overhead is involved in their creation and use. If a temporary mark is used after the text it points to has been modified results will be unpredictable. Permanent marks continue to point between the same two characters regardless of insertions and deletions made before or after them.

There are two different kinds of permanent marks which differ only in their behavior when text is inserted at the position of the mark; text is inserted to the left of a left-inserting mark and to the right of right-inserting mark.

6.2.2.2. Mark Functions

[Function]

markp mark

Description:

This function returns t if mark is a mark object, otherwise nil.

[Function]

mark-line mark

Description:

This function returns the line to which mark points.

[Function]

mark-charpos mark

Description:

This function returns the character position in the line of the character after mark, i.e. the number of characters before the mark in the mark's line.

[Function]

mark-buffer mark

Description:

Returns the buffer containing this mark.

[Function]

mark-absolute-position mark

Description:

This function returns the character position in the buffer of the character after the mark, i.e. the number of characters before the mark in the mark's buffer.

[Function]

mark-kind mark

Description:

This function returns one of :right-inserting, :left-inserting or :temporary depending on the mark's kind. A corresponding setf form changes the mark's kind.

[Function]

previous-character mark

Description:

[Function]

next-character mark

Description:

This function returns the character immediately before (after) the position of the mark, or nil if there is no previous (next) character. These characters may be set with setf when they exist; the setf methods for these forms signal errors when there is no previous or next character.

6.2.2.3. Making Marks

[Function]

mark line charpos &optional kind

Description:

This function returns a mark object that points to the charpos'th character of the line. Kind is the kind of mark to create, one of :temporary, :left-inserting, or :right-inserting. The default is :temporary.

[Function]

copy-mark mark &optional kind

Description:

This function returns a new mark pointing to the same position and of the same kind, or of kind kind if it is supplied.

[Function]

delete-mark mark

Description:

This function deletes mark. Delete any permanent marks when you are finished using it.

[Macro]

with-mark ({(mark pos [kind])}*) {form}*

Description:

This macro binds to each variable mark a mark of kind kind, which defaults to :temporary, pointing to the same position as the markpos. On exit from the scope the mark is deleted. The value of the last form is the value returned.

6.2.2.4. Moving Marks

These functions destructively modify marks to point to new positions. Other sections of this document describe mark moving routines specific to higher level text forms than characters and lines, such as words, sentences, paragraphs, Lisp forms, etc.

[Function]

move-to-position mark charpos &optional line

Description:

This function changes the mark to point to the given character position on the line line. Line defaults to mark's line.

[Function]

move-to-absolute-position mark position

Description:

This function changes the mark to point to the given character position in the buffer.

[Function]

move-mark mark new-position

Description:

This function moves mark to the same position as the mark new-position and returns it.

[Function]

line-start mark &optional line

Description:

[Function]

line-end mark &optional line

Description:

This function changes mark to point to the beginning or the end of line and returns it. Line defaults to mark's line.

[Function]

buffer-start mark &optional buffer

Description:

[Function]

buffer-end mark &optional buffer

Description:

These functions change mark to point to the beginning or end of buffer, which defaults to the buffer mark currently points into. If buffer is unsupplied, then it is an error for mark to be disassociated from any buffer.

[Function]

mark-before mark

Description:

[Function]

mark-after mark

Description:

These functions change mark to point one character before or after the current position. If there is no character before/after the current position, then they return nil and leave mark unmodified.

[Function]

character-offset mark n

Description:

This function changes mark to point n characters after (n before if n is negative) the current position. If there are less than n characters after (before) the mark, then this returns nil and mark is unmodified.

[Function]

line-offset mark n &optional charpos

Description:

This function changes mark to point n lines after (n before if n is negative) the current position. The character position of the resulting mark is (min (line-length resulting-line) (mark-charpos mark)) if charpos is unspecified, or (min (line-length resulting-line) charpos) if it is. As with character-offset, if there are not n lines then nil is returned and mark is not modified.

6.2.3. Regions

A region is simply a pair of marks: a starting mark and an ending mark. The text in a region consists of the characters following the starting mark and preceding the ending mark (keep in mind that a mark points between characters on a line, not at them). By modifying the starting or ending mark in a region it is possible to produce regions with a start and end which are out of order or even in different buffers. The use of such regions is undefined and may result in arbitrarily bad behavior.

6.2.3.1. Region Functions

[Function]

region start end

Description:

This function returns a region constructed from the marks start and end. It is an error for the marks to point to non-contiguous lines or for start to come after end.

[Function]

regionp region

Description:

This function returns t if region is a region object, otherwise nil.

[Function]

make-empty-region

Description:

This function returns a region with start and end marks pointing to the start of one empty line. The start mark is a :right-inserting mark, and the end is a :left-inserting mark.

[Function]

copy-region region

Description:

This function returns a region containing a copy of the text in the specified region. The resulting region is completely disjoint from region with respect to data references --- marks, lines, text, etc.

[Function]

region-to-string region

Description:

[Function]

string-to-region string

Description:

These functions coerce regions to Lisp strings and vice versa. Within the string, lines are delimited by newline characters.

[Function]

line-to-region line

Description:

This function returns a region containing all the characters on line. The first mark is :right-inserting and the last is :left-inserting.

[Function]

region-start region

Description:

[Function]

region-end region

Description:

This function returns the start or end mark of region.

[Function]

region-bounds region

Description:

This function returns as multiple-values the starting and ending marks of region.

[Function]

set-region-bounds region start end

Description:

This function sets the start and end of region to start and end. It is an error for start to be after or in a different buffer from end.

[Function]

count-lines region

Description:

This function returns the number of lines in the region, first and last lines inclusive. A newline is associated with the line it follows, thus a region containing some number of non-newline characters followed by one newline is one line, but if a newline were added at the beginning, it would be two lines.

[Function]

count-characters region

Description:

This function returns the number of characters in a given region. This counts line breaks as one character.

6.3. Buffers

A buffer is an object consisting of:

  1. A name.

  2. A piece of text.

  3. The insertion point.

  4. An associated file (optional).

  5. A write protect flag.

  6. Some variables.

  7. Some key bindings.

  8. A collection of modes.

  9. A list of modeline fields (optional).

Because of the way Hemlock is currently integrated in Cocoa, all modifications to buffer contents must take place in the GUI thread. Hemlock commands always run in the GUI thread, so most of the time you do not need to worry about it. If you are running code in another thread that needs to modify a buffer, you should perform that action using gui::execute-in-gui or gui::queue-for-gui.

There are no intrinsic limitations on examining buffers from any thread, however, Hemlock currently does no locking, so you risk seeing the buffer in an inconsistent state if you look at it outside the GUI thread.

6.3.1. The Current Buffer

Hemlock has the concept of the "current buffer". The current buffer is defined during Hemlock commands as the buffer of the hemlock view that received the key events that invoked the command. Many hemlock function operate on the current buffer rather than taking an explicit buffer argument. In effect, the current buffer is an implicit argument to many text manipulation functions.

[Function]

current-buffer

Description:

Returns the current buffer, which, during command execution, is the buffer that is the target of the command.

[Function]

current-point

Description:

This function returns the buffer-point of the current buffer . This is such a common idiom in commands that it is defined despite its trivial implementation.

[Function]

current-point-collapsing-selection

Description:

This function returns the buffer-point of the current buffer, after first deactivating any active region.

[Function]

current-point-extending-selection

Description:

This function returns the buffer-point of the current buffer, after first making sure there is an active region - if the region is already active, keeps it active, otherwise it establishes a new (empty) region at point.

[Function]

current-point-for-insertion

Description:

This function checks to see if the current buffer can be modified at its current point, and errors if not. Otherwise, it deletes the current selection if any, and returns the current point.

[Function]

current-point-for-deletion

Description:

This function checks to see if the current buffer can be modified at its current point and errors if not. Otherwise, if there is a section in the current buffer, it deletes it and returns NIL. If there is no selection, it returns the current point.

[Function]

current-point-unless-selection

Description:

This function checks to see if the current buffer can be modified at its current point and errors if not. Otherwise, if there's a selection in the current buffer, returns NIL. If there is no selection, it returns the current point.

[Function]

current-mark

Description:

This function returns the top of the current buffer's mark stack. There always is at least one mark at the beginning of the buffer's region, and all marks returned are right-inserting.

[Function]

pop-buffer-mark

Description:

This function pops the current buffer's mark stack, returning the mark. If the stack becomes empty, this pushes a new mark on the stack pointing to the buffer's start. This always deactivates the current region (see Active Regions).

[Function]

push-buffer-mark mark &optional activate-region

Description:

This function pushes mark into the current buffer's mark stack, ensuring that the mark is right-inserting. If mark does not point into the current buffer, this signals an error. Optionally, the current region is made active, but this never deactivates the current region (see Active Regions). Mark is returned.

[Function]

push-new-buffer-mark mark &optional activate-region

Description:

This function pushes a new mark onto the mark stack, at the position of mark. It's equivalent to calling push-buffer-mark on (copy-mark mark).

[Function]

all-buffers

Description:

This function returns a list of all the buffer objects made with make-buffer.

[Variable]

*buffer-names*

Description:

This variable holds a string-table mapping the name of a buffer to the corresponding buffer object.

6.3.2. Buffer Functions

[Function]

make-buffer name &key :modes :modeline-fields :delete-hook

Description:

[Hemlock Variable]

Default Modeline Fields

Description:

[Hemlock Variable]

Make Buffer Hook

Description:

make-buffer creates and returns a buffer with the given name. If a buffer named name already exists, nil is returned. Modes is a list of modes which should be in effect in the buffer, major mode first, followed by any minor modes. If this is omitted then the buffer is created with the list of modes contained in Default Modes. Modeline-fields is a list of modeline-field objects (see the Modelines section) which may be nil. delete-hook is a list of delete hooks specific to this buffer, and delete-buffer invokes these along with Delete Buffer Hook.

Buffers created with make-buffer are entered into the list (all-buffers), and their names are inserted into the string-table *buffer-names*. When a buffer is created the hook Make Buffer Hook is invoked with the new buffer.

[Function]

bufferp buffer

Description:

Returns t if buffer is a buffer object, otherwise nil.

[Function]

buffer-name buffer

Description:

[Hemlock Variable]

Buffer Name Hook

Description:

buffer-name returns the name, which is a string, of the given buffer. The corresponding setf method invokes Buffer Name Hook with buffer and the new name and then sets the buffer's name. When the user supplies a name for which a buffer already exists, the setf method signals an error.

[Function]

buffer-region buffer

Description:

Returns the buffer's region. Note this is the region that contains all the text in a buffer, as opposed to the current-region.

This can be set with setf to replace the buffer's text.

[Function]

buffer-pathname buffer

Description:

[Hemlock Variable]

Buffer Pathname Hook

Description:

buffer-pathname returns the pathname of the file associated with the given buffer, or nil if it has no associated file. This is the truename of the file as of the most recent time it was read or written. There is a setf form to change the pathname. When the pathname is changed the hook Buffer Pathname Hook is invoked with the buffer and new value.

[Function]

buffer-write-date buffer

Description:

Returns the write date for the file associated with the buffer in universal time format. When this the buffer-pathname is set, use setf to set this to the corresponding write date, or to nil if the date is unknown or there is no file.

[Function]

buffer-point buffer

Description:

Returns the mark which is the current location within buffer. To move the point, use move-mark or move-to-position

[Function]

buffer-mark buffer

Description:

This function returns the top of buffer's mark stack. There always is at least one mark at the beginning of buffer's region, and all marks returned are right-inserting.

[Function]

buffer-start-mark buffer

Description:

[Function]

buffer-end-mark buffer

Description:

These functions return the start and end marks of buffer's region:


(buffer-start-mark buffer )  &lt;==&gt;  (region-start (buffer-region buffer))

and


(buffer-end-mark buffer )  &lt;==&gt;  (region-end (buffer-region buffer))

[Function]

buffer-writable buffer

Description:

[Hemlock Variable]

Buffer Writable Hook

Description:

This function returns t if you can modify the buffer, nil if you cannot. If a buffer is not writable, then any attempt to alter text in the buffer results in an error. There is a setf method to change this value. The setf method invokes the functions in Buffer Writable Hook on the buffer and new value before storing the new value.

[Function]

buffer-modified buffer

Description:

[Hemlock Variable]

Buffer Modified Hook

Description:

buffer-modified returns t if the buffer has been modified, nil if it hasn't. This attribute is set whenever a text-altering operation is performed on a buffer. There is a setf method to change this value. The setf method invokes the functions in Buffer Modified Hook with the buffer whenever the value of the modified flag changes.

[Macro]

with-writable-buffer (buffer) &body body

Description:

This macro executes forms with buffer's writable status set. After forms execute, this resets the buffer's writable and modified status.

[Function]

buffer-signature buffer

Description:

This function returns an arbitrary number which reflects the buffer's current signature. The result is eql to a previous result if and only if the buffer has not been modified between the calls.

[Function]

buffer-variables buffer

Description:

This function returns a string-table containing the names of the buffer's local variables.

[Function]

buffer-modes buffer

Description:

This function returns the list of the names of the modes active in buffer. The major mode is first, followed by any minor modes. See the Modes chapter.

[Function]

buffer-delete-hook buffer

Description:

This function returns the list of buffer specific functions delete-buffer invokes when deleting a buffer . This is setf-able.

[Function]

delete-buffer buffer

Description:

[Hemlock Variable]

Delete Buffer Hook

Description:

delete-buffer removes buffer from (all-buffers) and its name from *buffer-names*. Before buffer is deleted, this invokes the functions on buffer returned by buffer-delete-hook and those found in Delete Buffer Hook. If buffer is the current-buffer, or if it is displayed in any view, then this function signals an error.

6.3.3. Modelines

A Buffer may specify a modeline, a line of text which is displayed across the bottom of a view to indicate status information. Modelines are described by a list of modeline-field objects which have individual update functions and are optionally fixed-width. These have an eql name for convenience in referencing and updating, but the name must be unique for all created modeline-field objects. All modeline-field functions must take a buffer as an argument and return a string. When displaying a modeline-field with a specified width, the result of the update function is either truncated or padded on the right to meet the constraint.

Whenever one of the following changes occurs, all of a buffer's modeline fields are updated:

  • A buffer's major mode is set.

  • One of a buffer's minor modes is turned on or off.

  • A buffer is renamed.

  • A buffer's pathname changes.

  • A buffer's modified status changes.

The policy is that whenever one of these changes occurs, it is guaranteed that the modeline will be updated before the next trip through redisplay. Furthermore, since the system cannot know what modeline-field objects the user has added whose update functions rely on these values, or how he has changed Default Modeline Fields, we must update all the fields.

The user should note that modelines can be updated at any time, so update functions should be careful to avoid needless delays (for example, waiting for a local area network to determine information).

[Function]

make-modeline-field &key :name :width :function

Description:

This function returns a modeline-field object with name, width, and function. Width defaults to nil meaning that the field is variable width; otherwise, the programmer must supply this as a positive integer. Function must take a buffer as an arguments and return a string. If name already names a modeline-field object, then this signals an error.

[Function]

modeline-field-name modeline-field

Description:

This function returns the name field of a modeline-field object. If this is set with setf, and the new name already names a modeline-field, then the setf method signals an error.

[Function]

modeline-field-p modeline-field

Description:

This function returns t if its argument is a modeline-field object, nil otherwise.

[Function]

modeline-field name

Description:

This returns the modeline-field object named name. If none exists, this returns nil.

[Function]

modeline-field-function modeline-field

Description:

Returns the function called when updating the modeline-field. When this is set with setf, the setf method updates modeline-field for all views on all buffers that contain the given field, so the next trip through redisplay will reflect the change. All modeline-field functions must take a buffer as an argument and return a string.

[Function]

modeline-field-width modeline-field

Description:

Returns the width to which modeline-field is constrained, or nil indicating that it is variable width. When this is set with setf, the setf method updates all modeline-fields for all views on all buffers that contain the given field, so the next trip through redisplay will reflect the change.

[Function]

buffer-modeline-fields buffer

Description:

Returns a copy of the list of buffer's modeline-field objects. This list can be destructively modified without affecting display of buffer's modeline, but modifying any particular field's components (for example, width or function) causes the changes to be reflected the next trip through redisplay in every modeline display that uses the modified modeline-field. When this is set with setf, the setf method method updates all modeline-fields on all views on the buffer, so next trip through the redisplay will reflect the change.

[Function]

buffer-modeline-field-p buffer field

Description:

If field, a modeline-field or the name of one, is in buffer's list of modeline-field objects, it is returned; otherwise, this returns nil.

[Function]

update-modeline-fields buffer

Description:

Arranges so that the modeline display is updated with the latest values at the end of current command.

6.4. Altering and Searching Text

A note on marks and text alteration: :temporary marks are invalid after any change has been made to the buffer the mark points to; it is an error to use a temporary mark after such a change has been made.

If text is deleted which has permanent marks pointing into it then they are left pointing to the position where the text was.

6.4.1. Altering Text

[Function]

insert-character mark character

Description:

[Function]

insert-string mark string

Description:

[Function]

insert-region mark region

Description:

Inserts character, string or region at mark. insert-character signals an error if character is not string-char-p. If string or region is empty, and mark is in some buffer, then Hemlock leaves buffer-modified of mark's buffer unaffected.

[Function]

ninsert-region mark region

Description:

Like insert-region, inserts the region at the mark's position, destroying the source region. This must be used with caution, since if anyone else can refer to the source region bad things will happen. In particular, one should make sure the region is not linked into any existing buffer. If region is empty, and mark is in some buffer, then Hemlock leaves buffer-modified of mark's buffer unaffected.

[Function]

delete-characters mark n

Description:

This deletes n characters after the mark (or -n before if n is negative). If n characters after (or -n before) the mark do not exist, then this returns nil; otherwise, it returns t. If n is zero, and mark is in some buffer, then Hemlock leaves buffer-modified of mark's buffer unaffected.

[Function]

delete-region region

Description:

This deletes region. This is faster than delete-and-save-region (below) because no lines are copied. If region is empty and contained in some buffer's buffer-region, then Hemlock leaves buffer-modified of the buffer unaffected.

[Function]

delete-and-save-region region

Description:

This deletes region and returns a region containing the original region's text. If region is empty and contained in some buffer's buffer-region, then Hemlock leaves buffer-modified of the buffer unaffected. In this case, this returns a distinct empty region.

[Function]

filter-region function region

Description:

Destructively modifies region by replacing the text of each line with the result of the application of function to a string containing that text. Function must obey the following restrictions:

  1. The argument may not be destructively modified.

  2. The return value may not contain newline characters.

  3. The return value may not be destructively modified after it is returned from function.

The strings are passed in order.

Using this function, a region could be uppercased by doing: (filter-region #'string-upcase region)

6.4.2. Text Predicates

[Function]

start-line-p mark

Description:

Returns t if the mark points before the first character in a line, nil otherwise.

[Function]

end-line-p mark

Description:

Returns t if the mark points after the last character in a line and before the newline, nil otherwise.

empty-line-p mark [Function] Return t if the line which mark points to contains no characters.

[Function]

blank-line-p line

Description:

Returns t if line contains only characters with a Whitespace attribute of 1. See the Character Attributes chapter for discussion of character attributes.

[Function]

blank-before-p mark

Description:

[Function]

blank-after-p mark

Description:

These functions test if all the characters preceding or following mark on the line it is on have a Whitespace attribute of 1.

[Function]

same-line-p mark1 mark2

Description:

Returns t if mark1 and mark2 point to the same line, or nil otherwise; That is,


(same-line-p a b) &lt;==&gt; (eq (mark-line a) (mark-line b))

[Function]

mark< mark1 mark2

Description:

[Function]

mark<= mark1 mark2

Description:

[Function]

mark= mark1 mark2

Description:

[Function]

mark/= mark1 mark2

Description:

[Function]

mark>= mark1 mark2

Description:

[Function]

mark> mark1 mark2

Description:

These predicates test the relative ordering of two marks in a piece of text, that is a mark is mark> another if it points to a position after it. An error is signalled if the marks do not point into the same buffer, except that for such marks mark= is always false and mark/= is always true.

[Function]

line< line1 line2

Description:

[Function]

line<= line1 line2

Description:

[Function]

line>= line1 line2

Description:

[Function]

line> line1 line2

Description:

These predicates test the ordering of line1 and line2. An error is signalled if the lines are not in the same buffer.

[Function]

lines-related line1 line2

Description:

This function returns t if line1 and line2 are in the same buffer, nil nil otherwise.

[Function]

first-line-p mark

Description:

[Function]

last-line-p mark

Description:

first-line-p returns t if there is no line before the line mark is on, and nil otherwise. Last-line-p similarly tests tests whether there is no line after mark.

6.4.3. Kill Ring

There is a global ring of regions deleted from buffers. Some commands save affected regions on the kill ring before performing modifications. You should consider making the command undoable, but this is a simple way of achieving a less satisfactory means for the user to recover.

[Function]

kill-region region current-type

Description:

This kills region saving it in the kill ring. Current-type is either :kill-forward or :kill-backward. When the last-command-type is one of these, this adds region to the beginning or end, respectively, of the top of the kill ring. The result of calling this is undoable using the command Undo (see the Hemlock User's Manual). This sets last-command-type to current-type, and it interacts with kill-characters.

[Function]

kill-characters mark count

Description:

[Hemlock Variable]

Character Deletion Threshold (initial value 5)

Description:

kill-characters kills count characters after mark if count is positive, otherwise before mark if count is negative. When count is greater than or equal to Character Deletion Threshold, the killed characters are saved on the kill ring. This may be called multiple times contiguously (that is, without last-command-type being set) to accumulate an effective count for purposes of comparison with the threshold.

This sets last-command-type, and it interacts with kill-region. When this adds a new region to the kill ring, it sets last-command-type to :kill-forward (if count is positive) or :kill-backward (if count is negative). When last-command-type is :kill-forward or :kill-backward, this adds the killed characters to the beginning (if count is negative) or the end (if count is positive) of the top of the kill ring, and it sets last-command-type as if it added a new region to the kill ring. When the kill ring is unaffected, this sets last-command-type to :char-kill-forward or :char-kill-backward depending on whether count is positive or negative, respectively.

This returns mark if it deletes characters. If there are not count characters in the appropriate direction, this returns nil.

6.4.4. Active Regions

Every buffer has a mark stack and a mark known as the point where most text altering nominally occurs. Between the top of the mark stack, the current-mark, and the current-buffer's point, the current-point, is what is known as the current-region . Certain commands signal errors when the user tries to operate on the current-region without its having been activated. If the user turns off this feature, then the current-region is effectively always active.

When writing a command that marks a region of text, the programmer should make sure to activate the region. This typically occurs naturally from the primitives that you use to mark regions, but sometimes you must explicitly activate the region. These commands should be written this way, so they do not require the user to separately mark an area and then activate it. Commands that modify regions do not have to worry about deactivating the region since modifying a buffer automatically deactivates the region. Commands that insert text often activate the region ephemerally; that is, the region is active for the immediately following command, allowing the user wants to delete the region inserted, fill it, or whatever.

Once a marking command makes the region active, it remains active until:

  • a command uses the region,

  • a command modifies the buffer,

  • a command changes the current window or buffer,

  • a command signals an editor-error,

  • or the user types C-g.

[Hemlock Variable]

Active Regions Enabled (initial value t)

Description:

When this variable is non-nil, some primitives signal an editor-error if the region is not active. This may be set to nil for more traditional Emacs region semantics.

[Variable]

*ephemerally-active-command-types*

Description:

This is a list of command types, and its initial value is the list of :ephemerally-active and :unkill. When the previous command's type is one of these, the current-region is active for the currently executing command only, regardless of whether it does something to deactivate the region. However, the current command may activate the region for future commands. :ephemerally-active is a default command type that may be used to ephemerally activate the region, and:unkill is the type used by two commands, Un-kill and Rotate Kill Ring (what users typically think of as C-y and M-y).

[Function]

activate-region

Description:

This makes the current-region active.

[Function]

deactivate-region

Description:

After invoking this the current-region is no longer active.

[Function]

region-active-p

Description:

Returns whether the current-region is active, including ephemerally. This ignores Active Regions Enabled.

[Function]

check-region-active

Description:

This signals an editor-error when active regions are enabled, and the current-region is not active.

[Function]

current-region &optional error-if-not-active deactivate-region

Description:

This returns a region formed with current-mark and current-point, optionally signaling an editor-error if the current region is not active. Error-if-not-active defaults to t. Each call returns a distinct region object. Depending on deactivate-region (defaults to t), fetching the current region deactivates it. Hemlock primitives are free to modify text regardless of whether the region is active, so a command that checks for this can deactivate the region whenever it is convenient.

6.4.5. Searching and Replacing

Before using any of these functions to do a character search, look at character attributes. They provide a facility similar to the syntax table in real Emacs. Syntax tables are a powerful, general, and efficient mechanism for assigning meanings to characters in various modes.

[Function]

new-search-pattern kind direction pattern &optional result-search-pattern

Description:

Returns a search-pattern object which can be given to the find-pattern and replace-pattern functions. A search-pattern is a specification of a particular sort of search to do. direction is either :forward or :backward, indicating the direction to search in. kind specifies the kind of search pattern to make, and pattern is a thing which specifies what to search for. The interpretation of pattern depends on the kind of pattern being made. Currently defined kinds of search pattern are:

:string-insensitive--- Does a case-insensitive string search for pattern

:string-sensitive--- Does a case-sensitive string search for pattern.

:character--- Finds an occurrence of the character pattern. This is case sensitive.

:not-character--- Find a character which is not the character pattern.

:test--- Finds a character which satisfies the function pattern. This function may not be applied an any particular fashion, so it should depend only on what its argument is, and should have no side-effects.

:test-not--- Similar to :test, except it finds a character that fails the test.

:any--- Finds a character that is in the string pattern.

:not-any--- Finds a character that is not in the string pattern.

result-search-pattern, if supplied, is a search-pattern to destructively modify to produce the new pattern. Where reasonable this should be supplied, since some kinds of search patterns may involve large data structures.

[Function]

search-pattern-p search-pattern

Description:

Returns t if search-pattern is a search-pattern object, otherwise nil.

[Function]

get-search-pattern string direction

Description:

get-search-pattern interfaces to a default search string and pattern that search and replacing commands can use. These commands then share a default when prompting for what to search or replace, and save on consing a search pattern each time they execute. This uses Default Search Kind (see the Hemlock User's Manual) when updating the pattern object.

[Variable]

*last-search-string*

Description:

Returns the last string searched for, useful when prompting.

[Function]

find-pattern mark search-pattern

Description:

Find the next match of search-pattern starting at mark. If a match is found then mark is altered to point before the matched text and the number of characters matched is returned. If no match is found then nil is returned and mark is not modified.

[Function]

replace-pattern mark search-pattern replacement &optional n

Description:

Replace n matches of search-pattern with the string replacement starting at mark. If n is nil (the default) then replace all matches. A mark pointing before the last replacement done is returned.

6.5. The Current Environment

6.5.1. Different Scopes

In Hemlock the "current" values of variables, key bindings and character-attributes depend on the current buffer and the modes active in it. There are three possible scopes for Hemlock values:

buffer local

The value is present only if the buffer it is local to is the current buffer.

mode local

The value is present only when the mode it is local to is active in the current buffer.

global

The value is always present unless shadowed by a buffer or mode local value.

6.5.2. Shadowing

It is possible that there are different values for the same thing in in different scopes. For example, there be might a global binding for a given variable and also a local binding in the current buffer. Whenever there is a conflict, shadowing occurs, permitting only one of the values to be visible in the current environment.

The process of resolving such a conflict can be described as a search down a list of places where the value might be defined, returning the first value found. The order for the search is as follows:

  1. Local values in the current buffer.

  2. Mode local values in the minor modes of the current buffer, in order from the highest precedence mode to the lowest precedence mode. The order of minor modes with equal precedences is undefined.

  3. Mode local values in the current buffer's major mode.

  4. Global values.

6.6. Hemlock Variables

Hemlock implements a system of variables separate from normal Lisp variables for the following reasons:

  1. Hemlock has different scoping rules which are useful in an editor. Hemlock variables can be local to a buffer or a mode.

  2. Hemlock variables have hooks, lists of functions called when someone sets the variable. See variable-value for the arguments Hemlock passes to these hook functions.

  3. There is a database of variable names and documentation which makes it easier to find out what variables exist and what their values mean.

6.6.1. Variable Names

To the user, a variable name is a case insensitive string. This string is referred to as the string name of the variable. A string name is conventionally composed of words separated by spaces.

In Lisp code a variable name is a symbol. The name of this symbol is created by replacing any spaces in the string name with hyphens. This symbol name is always interned in the Hemlock package.

[Variable]

*global-variable-names*

Description:

This variable holds a string-table of the names of all the global Hemlock variables. The value of each entry is the symbol name of the variable.

[Function]

current-variable-tables

Description:

This function returns a list of variable tables currently established, globally, in the current buffer, and by the modes of the current-buffer. This list is suitable for use with prompt-for-variable.

6.6.2. Variable Functions

In the following descriptions name is the symbol name of the variable.

[Function]

defhvar string-name documentation &key :mode :buffer :hooks :value

Description:

This function defines a Hemlock variable. Functions that take a variable name signal an error when the variable is undefined.

string-name--- The string name of the variable to define.

documentation--- The documentation string for the variable.

:mode, :buffer--- If buffer is supplied, the variable is local to that buffer. If mode is supplied, it is local to that mode. If neither is supplied, it is global.

:value--- This is the initial value for the variable, which defaults to nil.

:hooks--- This is the initial list of functions to call when someone sets the variable's value. These functions execute before Hemlock establishes the new value. See variable-value for the arguments passed to the hook functions.

If a variable with the same name already exists in the same place, then defhvar sets its hooks and value from hooks and value if the user supplies these keywords.

[Function]

variable-value name &optional kind where

Description:

This function returns the value of a Hemlock variable in some place. The following values for kind are defined:

:current--- Return the value present in the current environment, taking into consideration any mode or buffer local variables. This is the default.

:global--- Return the global value.

:mode--- Return the value in the mode named where.

:buffer--- Return the value in the buffer where.

When set with setf, Hemlock sets the value of the specified variable and invokes the functions in its hook list with name, kind, where, and the new value.

[Function]

variable-documentation name &optional kind where

Description:

[Function]

variable-hooks name &optional kind where

Description:

[Function]

variable-name name &optional kind where

Description:

These function return the documentation, hooks and string name of a Hemlock variable. The kind and where arguments are the same as for variable-value. The documentation and hook list may be set using setf.

[Function]

string-to-variable string

Description:

This function converts a string into the corresponding variable symbol name. String need not be the name of an actual Hemlock variable.

[Macro]

value name

Description:

[Macro]

setv name new-value

Description:

These macros get and set the current value of the Hemlock variable name. Name is not evaluated. There is a setf form for value.

[Macro]

hlet ({(var value)}*){form}*

Description:

This macro is very similar to let in effect; within its scope each of the Hemlock variables var have the respective values, but after the scope is exited by any means the binding is removed. This does not cause any hooks to be invoked. The value of the last form is returned.

[Function]

hemlock-bound-p name &optional kind where

Description:

Returns t if name is defined as a Hemlock variable in the place specified by kind and where, or nil otherwise.

[Function]

delete-variable name &optional kind where

Description:

[Hemlock Variable]

Delete Variable Hook

Description:

delete-variable makes the Hemlock variable name no longer defined in the specified place. Kind and where have the same meanings as they do for variable-value, except that :current is not available, and the default for kind is :global

An error will be signaled if no such variable exists. The hook, Delete Variable Hook is invoked with the same arguments before the variable is deleted.

6.6.3. Hooks

Hemlock actions such as setting variables, changing buffers, changing windows, turning modes on and off, etc., often have hooks associated with them. A hook is a list of functions called before the system performs the action. The manual describes the object specific hooks with the rest of the operations defined on these objects.

Often hooks are stored in Hemlock variables, Delete Buffer Hook and Set Window Hook for example. This leads to a minor point of confusion because these variables have hooks that the system executes when someone changes their values. These hook functions Hemlock invokes when someone sets a variable are an example of a hook stored in an object instead of a Hemlock variable. These are all hooks for editor activity, but Hemlock keeps them in different kinds of locations. This is why some of the routines in this section have a special interpretation of the hook place argument.

[Macro]

add-hook place hook-fun

Description:

[Macro]

remove-hook place hook-fun

Description:

These macros add or remove a hook function in some place. If hook-fun already exists in place, this call has no effect. If place is a symbol, then it is a Hemlock variable; otherwise, it is a generalized variable or storage location. Here are two examples:


(add-hook delete-buffer-hook 'remove-buffer-from-menu)
(add-hook (variable-hooks 'check-mail-interval)
          'reschedule-mail-check)

[Macro]

invoke-hook place &rest args

Description:

This macro calls all the functions in place. If place is a symbol, then it is a Hemlock variable; otherwise, it is a generalized variable.

6.7. Commands

6.7.1. Introduction

The way that the user tells Hemlock to do something is by invoking a command. Commands have three attributes:

name

A command's name provides a way to refer to it. Command names are usually capitalized words separated by spaces, such as Forward Word.

documentation

The documentation for a command is used by on-line help facilities.

function

A command is implemented by a Lisp function, which is callable from Lisp.

6.7.1.1. Defining Commands

[Variable]

*command-names*

Description:

Holds a string-table associating command names to command objects. Whenever a new command is defined it is entered in this table.

[Macro]

defcommand {command-name | (command-name function-name &key)} lambda-list command-doc {function-doc} {form}*

Description:

Defines a command named name. defcommand creates a function to implement the command from the lambda-list and forms supplied. The lambda-list must specify one required argument, see below, which by convention is typically named p. If the caller does not specify function-name, defcommand creates the command name by replacing all spaces with hyphens and appending "-command". Any keyword arguments are as for make-command. Command-doc becomes the command documentation for the command. Function-doc, if present, becomes the documentation for the function and should primarily describe issues involved in calling the command as a function, such as what any additional arguments are.

[Function]

make-command name documentation function &key :transparent-p

Description:

Defines a new command named name, with command documentation documentation and function function. If :transparent-p is true, the command becomes transparent. The command in entered in the string-table *command-names*, with the command object as its value. Normally command implementors will use the defcommand macro, but this permits access to the command definition mechanism at a lower level, which is occasionally useful.

[Function]

commandp command

Description:

Returns t if command is a command object, otherwise nil.

[Function]

command-documentation command

Description:

[Function]

command-function command

Description:

[Function]

command-name command

Description:

Returns the documentation, function, or name for command. These may be set with setf.

6.7.1.2. Command Documentation

Command documentation is a description of what the command does when it is invoked as an extended command or from a key. Command documentation may be either a string or a function. If the documentation is a string then the first line should briefly summarize the command, with remaining lines filling the details. Example:


(defcommand "Forward Character" (p)
  "Move the point forward one character.
   With prefix argument move that many characters, with negative
   argument go backwards."
. . .)

Command documentation may also be a function of one argument. The function is called with either :short or :full, indicating that the function should return a short documentation string or do something to document the command fully.

6.7.2. The Command Interpreter

The command interpreter is the functionality invoked by the event handler to process key-events from the keyboard and dispatch to different commands on the basis of what the user types. When the command interpreter executes a command, we say it invokes the command. The command interpreter also provides facilities for communication between contiguously running commands, such as a last command type register. It also takes care of resetting communication mechanisms, clearing the echo area, displaying partial keys typed slowly by the user, etc.

6.7.2.1. Controlling The Command Interpreter

[Hemlock Variable]

Command Abort Hook

Description:

The command interpreter invokes the function in this variable whenever someone aborts a command (for example, if someone called editor-error).

6.7.2.2. Editor Input

The canonical representation of editor input is a key-event structure. Users can bind commands to keys, which are non-empty sequences of key-events. A key-event consists of an identifying token known as a keysym and a field of bits representing modifiers. Users define keysym names by supplying names that reflect the legends on their keyboard's keys. Users define modifier names similarly, but the system chooses the bit and mask for recognizing the modifier. You can use keysym and modifier names to textually specify key-events and Hemlock keys in a #k syntax. The following are some examples:


#k"C-u"
#k"Control-u"
#k"c-m-z"
#k"control-x meta-d"
#k"a"
#k"A"
#k"Linefeed"

This is convenient for use within code and in init files containing bind-key calls.

The #k syntax is delimited by double quotes. Within the double quotes, spaces separate multiple key-events. A single key-event optionally starts with modifier names terminated by hyphens. Modifier names are alphabetic sequences of characters which the system uses case-insensitively. Following modifiers is a keysym name, which is case-insensitive if it consists of multiple characters, but if the name consists of only a single character, then it is case-sensitive.

You can escape special characters---hyphen, double quote, open angle bracket, close angle bracket, and space---with a backslash, and you can specify a backslash by using two contiguously. You can use angle brackets to enclose a keysym name with many special characters in it. Between angle brackets appearing in a keysym name position, there are only two special characters, the closing angle bracket and backslash.

For more information on key-events see the Key-events section.

6.7.2.3. Binding Commands to Keys

The command interpreter determines which command to invoke on the basis of key bindings. A key binding is an association between a command and a sequence of key-events. A sequence of key-events is called a key and is represented by a single key-event or a sequence (list or vector) of key-events.

Since key bindings may be local to a mode or buffer, the current environment determines the set of key bindings in effect at any given time. When the command interpreter tries to find the binding for a key, it first checks if there is a local binding in the current buffer, then if there is a binding in each of the minor modes and the major mode for the current buffer, and finally checks to see if there is a global binding. If no binding is found, then the command interpreter beeps or flashes the screen to indicate this.

[Function]

bind-key name key &optional kind where

Description:

This function associates command name and key in some environment. Key is either a key-event or a sequence of key-events. There are three possible values of kind:

:global--- The default, make a global key binding.

:mode--- Make a mode specific key binding in the mode whose name is where.

:buffer--- Make a binding which is local to buffer where.

This processes key for key translations before establishing the binding.

If the key is some prefix of a key binding which already exists in the specified place, then the new one will override the old one, effectively deleting it.

do-alpha-key-events is useful for setting up bindings in certain new modes.

[Function]

command-bindings command

Description:

This function returns a list of the places where command is bound. A place is specified as a list of the key (always a vector), the kind of binding, and where (either the mode or buffer to which the binding is local, or nil if it is a global).

[Function]

delete-key-binding key &optional kind where

Description:

This function removes the binding of key in some place. Key is either a key-event or a sequence of key-events. kind is the kind of binding to delete, one of :global(the default), :mode or :buffer. If kind is :mode, where is the mode name, and if kind is :buffer, then where is the buffer.

This function signals an error if key is unbound.

This processes key for key translations before deleting the binding.

[Function]

get-command key &optional kind where

Description:

This function returns the command bound to key, returning nil if it is unbound. Key is either a key-event or a sequence of key-events. If key is an initial subsequence of some keys, then this returns the keyword :prefix. There are four cases of kind:

:current--- Return the current binding of key using the current buffer's search list. If there are any transparent key bindings for key, then they are returned in a list as a second value.

:global--- Return the global binding of key. This is the default.

:mode--- Return the binding of key in the mode named where.

:buffer--- Return the binding of key local to the buffer where.

This processes key for key translations before looking for any binding.

[Function]

map-bindings function kind &optional where

Description:

This function maps over the key bindings in some place. For each binding, this passes function the key and the command bound to it. Kind and where are the same as in bind-key. The key is not guaranteed to remain valid after a given iteration.

6.7.2.4. Key Translation

Key translation is a process that the command interpreter applies to keys before doing anything else. There are two kinds of key translations: substitution and bit-prefix. In either case, the command interpreter translates a key when a specified key-event sequence appears in a key.

In a substitution translation, the system replaces the matched subsequence with another key-event sequence. Key translation is not recursively applied to the substituted key-events.

In a bit-prefix translation, the system removes the matched subsequence and effectively sets the specified bits in the next key-event in the key.

While translating a key, if the system encounters an incomplete final subsequence of key-events, it aborts the translation process. This happens when those last key-events form a prefix of some translation. It also happens when they translate to a bit-prefix, but there is no following key-event to which the system can apply the indicated modifier. If there is a binding for this partially untranslated key, then the command interpreter will invoke that command; otherwise, it will wait for the user to type more key-events.

[Function]

key-translation key

Description:

This form is setf-able and allows users to register key translations that the command interpreter will use as users type key-events.

This function returns the key translation for key, returning nil if there is none. Key is either a key-event or a sequence of key-events. If key is a prefix of a translation, then this returns :prefix.

A key translation is either a key or modifier specification. The bits translations have a list form: (:bits {bit-name}*).

Whenever key appears as a subsequence of a key argument to the binding manipulation functions, that portion will be replaced with the translation.

6.7.2.5. Transparent Key Bindings

Key bindings local to a mode may be transparent. A transparent key binding does not shadow less local key bindings, but rather indicates that the bound command should be invoked before the first normal key binding. Transparent key bindings are primarily useful for implementing minor modes such as auto fill and word abbreviation. There may be several transparent key bindings for a given key, in which case all of the transparent commands are invoked in the order they were found. If there no normal key binding for a key typed, then the command interpreter acts as though the key is unbound even if there are transparent key bindings.

The :transparent-p argument to defmode determines whether all the key bindings in a mode are transparent or not. In addition a particular command may be declared to be transparent by the :transparent-p argument to defcommand and make-command.

6.7.3. Command Types

In many editors the behavior of a command depends on the kind of command invoked before it. Hemlock provides a mechanism to support this known as command type.

[Function]

last-command-type

Description:

This returns the command type of the last command invoked. If this is set with setf, the supplied value becomes the value of last-command-type until the next command completes. If the previous command did not set last-command-type, then its value is nil. Normally a command type is a keyword. The command type is not cleared after a command is invoked due to a transparent key binding.

6.7.4. Command Arguments

There are three ways in which a command may be invoked: It may be bound to a key which has been typed, it may be invoked as an extended command, or it may be called as a Lisp function. Ideally commands should be written in such a way that they will behave sensibly no matter which way they are invoked. The functions which implement commands must obey certain conventions about argument passing if the command is to function properly.

6.7.4.1. The Prefix Argument

Whenever a command is invoked it is passed as its first argument what is known as the prefix argument. The prefix argument is always either an integer or nil. When a command uses this value it is usually as a repeat count, or some conceptually similar function.

[Function]

prefix-argument

Description:

This function returns the current value of the prefix argument. When set with setf, the new value becomes the prefix argument for the next command. If the prefix argument is not set by the previous command then the prefix argument for a command is nil. The prefix argument is not cleared after a command is invoked due to a transparent key binding.

6.7.4.2. Lisp Arguments

It is often desirable to call commands from Lisp code, in which case arguments which would otherwise be prompted for are passed as optional arguments following the prefix argument. A command should prompt for any arguments not supplied.

6.8. Modes

A mode is a collection of Hemlock values which may be present in the current environment depending on the editing task at hand. An example of a typical mode is Lisp, for editing Lisp code.

6.8.1. Mode Hooks

When a mode is added to or removed from a buffer, its mode hook is invoked. The hook functions take two arguments, the buffer involved and t if the mode is being added or nil if it is being removed. Mode hooks are typically used to make a mode do something additional to what it usually does. One might, for example, make a Text mode hook that turned on auto-fill mode when you entered.

6.8.2. Major and Minor Modes

There are two kinds of modes, major modes and minor modes. A buffer always has exactly one major mode, but it may have any number of minor modes. Major modes may have mode character attributes while minor modes may not.

A major mode is usually used to change the environment in some major way, such as to install special commands for editing some language. Minor modes generally change some small attribute of the environment, such as whether lines are automatically broken when they get too long. A minor mode should work regardless of what major mode and minor modes are in effect.

[Hemlock Variable]

Default Modes (initial value '("Fundamental" "Save"))

Description:

This variable contains a list of mode names which are instantiated in a buffer when no other information is available.

[Variable]

*mode-names*

Description:

Holds a string-table of the names of all the modes.

[Command]

Illegal

Description:

This is a useful command to bind in modes that wish to shadow global bindings by making them effectively illegal. Also, although less likely, minor modes may shadow major mode bindings with this. This command calls editor-error.

6.8.3. Mode Functions

[Function]

defmode name &key :setup-function :cleanup-function :major-p :precedence :transparent-p :documentation

Description:

This function defines a new mode named name, and enters it in *mode-names*. If major-p is supplied and is not nil then the mode is a major mode; otherwise it is a minor mode.

Setup-function and cleanup-function are functions which are invoked with the buffer affected, after the mode is turned on, and before it is turned off, respectively. These functions typically are used to make buffer-local key or variable bindings and to remove them when the mode is turned off.

Precedence is only meaningful for a minor mode. The precedence of a minor mode determines the order in which it in a buffer's list of modes. When searching for values in the current environment, minor modes are searched in order, so the precedence of a minor mode determines which value is found when there are several definitions.

Transparent-p determines whether key bindings local to the defined mode are transparent. Transparent key bindings are invoked in addition to the first normal key binding found rather than shadowing less local key bindings.

Documentation is some introductory text about the mode. Commands such as Describe Mode use this.

[Function]

mode-documentation name

Description:

This function returns the documentation for the mode named name.

[Function]

buffer-major-mode buffer

Description:

[Hemlock Variable]

Buffer Major Mode Hook

Description:

buffer-major-mode returns the name of buffer's major mode. The major mode may be changed with setf; then Buffer Major Mode Hook is invoked with buffer and the new mode.

[Function]

buffer-minor-mode buffer name

Description:

[Hemlock Variable]

Buffer Minor Mode Hook

Description:

buffer-minor-mode returns t if the minor mode name is active in buffer, nil otherwise. A minor mode may be turned on or off by using setf; then Buffer Minor Mode Hook is invoked with buffer, name and the new value.

[Function]

mode-variables name

Description:

Returns the string-table of mode local variables.

[Function]

mode-major-p name

Description:

Returns t if name is the name of a major mode, or nil if it is the name of a minor mode. It is an error for name not to be the name of a mode.

6.9. Character Attributes

6.9.1. Introduction

Character attributes provide a global database of information about characters. This facility is similar to, but more general than, the syntax tables of other editors such as Emacs. For example, you should use character attributes for commands that need information regarding whether a character is whitespace or not. Use character attributes for these reasons:

  1. If this information is all in one place, then it is easy the change the behavior of the editor by changing the syntax table, much easier than it would be if character constants were wired into commands.

  1. This centralization of information avoids needless duplication of effort.

  1. The syntax table primitives are probably faster than anything that can be written above the primitive level.

Note that an essential part of the character attribute scheme is that character attributes are global and are there for the user to change. Information about characters which is internal to some set of commands (and which the user should not know about) should not be maintained as a character attribute. For such uses various character searching abilities are provided by the function find-pattern. 20).

6.9.2. Character Attribute Names

As for Hemlock variables, character attributes have a user visible string name, but are referred to in Lisp code as a symbol. The string name, which is typically composed of capitalized words separated by spaces, is translated into a keyword by replacing all spaces with hyphens and interning this string in the keyword package. The attribute named "Ada Syntax" would thus become :ada-syntax.

[Variable]

*character-attribute-names*

Description:

Whenever a character attribute is defined, its name is entered in this string-table, with the corresponding keyword as the value.

6.9.3. Character Attribute Functions

[Function]

defattribute name documentation &optional type initial-value

Description:

This function defines a new character attribute with name, a string. Character attribute operations take attribute arguments as a keyword whose name is name uppercased with spaces replaced by hyphens.

Documentation describes the uses of the character attribute.

Type, which defaults to (mod 2), specifies what type the values of the character attribute are. Values of a character attribute may be of any type which may be specified to make-array. Initial-value (default 0) is the value which all characters will initially have for this attribute.

[Function]

character-attribute-name attribute

Description:

[Function]

character-attribute-documentation attribute

Description:

These functions return the name or documentation for attribute.

[Function]

character-attribute attribute character

Description:

[Hemlock Variable]

Character Attribute Hook

Description:

character-attribute returns the value of attribute for character. This signals an error if attribute is undefined.

setf will set a character's attributes. This setf method invokes the functions in Character Attribute Hook on the attribute and character before it makes the change.

If character is nil, then the value of the attribute for the beginning or end of the buffer can be accessed or set. The buffer beginning and end thus become a sort of fictitious character, which simplifies the use of character attributes in many cases.

[Function]

character-attribute-p symbol

Description:

This function returns t if symbolis the name of a character attribute, nil otherwise.

[Function]

shadow-attribute attribute character value mode

Description:

[Hemlock Variable]

Shadow Attribute Hook

Description:

This function establishes value as the value of character's attribute attribute when in the mode mode. Mode must be the name of a major mode. Shadow Attribute Hook is invoked with the same arguments when this function is called. If the value for an attribute is set while the value is shadowed, then only the shadowed value is affected, not the global one.

[Function]

unshadow-attribute attribute character mode

Description:

[Hemlock Variable]

Unshadow Attribute Hook

Description:

Make the value of attribute for character no longer be shadowed in mode. Unshadow Attribute Hook is invoked with the same arguments when this function is called.

[Function]

find-attribute mark attribute &optional test

Description:

[Function]

reverse-find-attribute mark attribute &optional test

Description:

These functions find the next (or previous) character with some value for the character attribute attribute starting at mark. They pass test one argument, the value of attribute for the character tested. If the test succeeds, then these routines modify mark to point before (after for reverse-find-attribute) the character which satisfied the test. If no characters satisfy the test, then these return nil, and mark remains unmodified. Test defaults to #'not-zerop. There is no guarantee that the test is applied in any particular fashion, so it should have no side effects and depend only on its argument.

[Function]

find-not-attribute mark attribute

Description:

[Function]

reverse-find-not-attribute mark attribute

Description:

These are equivalent to (find-attribute mark attribute #'zerop) and (reverse-find-attribute mark attribute #'zerop), respectively.

6.9.4. Character Attribute Hooks

It is often useful to use the character attribute mechanism as an abstract interface to other information about characters which in fact is stored elsewhere. For example, some implementation of Hemlock might decide to define a Print Representation attribute which controls how a character is displayed on an output device.

To make this easy to do, each attribute has a list of hook functions which are invoked with the attribute, character and new value whenever the current value changes for any reason.

[Function]

character-attribute-hooks attribute

Description:

Return the current hook list for attribute. This may be set with setf. The add-hook and remove-hook macros should be used to manipulate these lists.

6.9.5. System Defined Character Attributes

These are predefined in Hemlock:

"Whitespace"

A value of 1 indicates the character is whitespace.

"Word Delimiter"

A value of 1 indicates the character separates words (see the English Text Buffers section).

"Space"

This is like Whitespace, but it should not include Newline. Hemlock uses this primarily for handling indentation on a line.

"Sentence Terminator"

A value of 1 indicates these characters terminate sentences (see the English Text Buffers section).

"Sentence Closing Char"

A value of 1 indicates these delimiting characters, such as " or ), may follow a Sentence Terminator.

"Paragraph Delimiter"

A value of 1 indicates these characters delimit paragraphs when they begin a line (see the English Text Buffers section).

"Page Delimiter"

A value of 1 indicates this character separates logical pages when it begins a line.

"Lisp Syntax"

This uses symbol values from the following:

  • nil These characters have no interesting properties.

  • :space These characters act like whitespace and should not include Newline.

  • :newline This is the Newline character.

  • :open-paren This is ( character.

  • :close-paren This is ) character.

  • :prefix This is a character that is a part of any form it precedes for example, the single quote, '.

  • :string-quote This is the character that quotes a string literal, ".

  • :char-quote This is the character that escapes a single character, \.

  • :comment This is the character that makes a comment with the rest of the line,;.

  • :constituent These characters are constitute symbol names.

6.10. Controlling the Display

6.10.1. Views

A hemlock-view represents the GUI object(s) used to display the contents of a buffer. Conceptually it consists of a text buffer, a modeline for semi-permanent status info, an echo area for transient status info, and a text input area for reading prompted input. (Currently the last two are conflated, i.e. text input happens in the echo area).

The API for working with hemlock-views is not fully defined yet. If you need to work with views beyond what's listed here, you will probably need to get in the sources and find some internal functions to call.

6.10.2. The Current View

[Function]

current-view

Description:

current-view returns the hemlock view which is the target of the currently executing command. This is usually the frontmost hemlock window in the current application.

6.10.3. View Functions

[Function]

hemlock-view-p object

Description:

This function returns t if object is a hemlock view, otherwise nil.

[Function]

hemlock-view-buffer view

Description:

This function returns the buffer which is displayed in the view.

6.10.4. Cursor Positions

[Function]

mark-column mark

Description:

This function returns the X position at which mark would be displayed, supposing its line was displayed on an infinitely wide screen. This takes into consideration strange characters such as tabs.

[Function]

move-to-column mark column &optional line

Description:

This function is analogous to move-to-position, except that it moves mark to the position on line which corresponds to the specified column. If the line would not reach to the specified column, then nil is returned and mark is not modified. Note that since a character may be displayed on more than one column on the screen, several different values of column may cause mark to be moved to the same position.

6.10.5. Redisplay

The display of the buffer contents on the screen is updated at the end of each command. The following function can be used to control the scroll position of the buffer in the view.

[Function]

set-scroll-position how &optional what

Description:

Normally, after a command that changes the contents of the buffer or the selection (i.e. the active region), the event handler repositions the view so that the selection is visible, scrolling the buffer as necessary. Calling this function tells the system to not do that, and instead to position the buffer in a particular way. how can be one of the following:

:center-selection--- This causes the selection (or the point) to be centered in the visible area. what is ignored.

:page-up--- This causes the previous page of the buffer to be shown what is ignored.

:page-down--- This causes the next page of the buffer to be shown. what is ignored.

:lines-up--- This causes what previous lines to be scrolled in at the top. what must be an integer.

:lines-down--- This causes what next lines to be scrolled in at the bottom. what must be an integer.

:line--- This causes the line containing what to be scrolled to the top of the view. what must be a mark.

6.11. Logical Key-Events

6.11.1. Introduction

Some primitives such as prompt-for-key and commands such as Emacs query replace read key-events directly from the keyboard instead of using the command interpreter. To encourage consistency between these commands and to make them portable and easy to customize, there is a mechanism for defininglogical key-events. A logical key-event is a keyword which stands for some set of key-events. The system globally interprets these key-events as indicators a particular action. For example, the :help logical key-event represents the set of key-events that request help in a given Hemlock implementation. This mapping is a many-to-many mapping, not one-to-one, so a given logical key-event may have multiple corresponding actual key-events. Also, any key-event may represent different logical key-events.

6.11.2. Logical Key-Event Functions

[Variable]

*logical-key-event-names*

Description:

This variable holds a string-table mapping all logical key-event names to the keyword identifying the logical key-event.

[Function]

define-logical-key-event string-name documentation

Description:

This function defines a new logical key-event with name string-name. Logical key-event operations take logical key-events arguments as a keyword whose name is string-name uppercased with spaces replaced by hyphens.

Documentation describes the action indicated by the logical key-event.

[Function]

logical-key-event-key-events keyword

Description:

This function returns the list of key-events representing the logical key-event keyword.

[Function]

logical-key-event-name keyword

Description:

[Function]

logical-key-event-documentation keyword

Description:

These functions return the string name and documentation given to define-logical-key-event for logical key-event keyword.

[Function]

logical-key-event-p key-event keyword

Description:

This function returns t if key-event is the logical key-event keyword. This is setf-able establishing or disestablishing key-events as particular logical key-events. It is a error for keyword to be an undefined logical key-event.

6.11.3. System Defined Logical Key-Events

There are many default logical key-events, some of which are used by functions documented in this manual. If a command wants to read a single key-event command that fits one of these descriptions then the key-event read should be compared to the corresponding logical key-event instead of explicitly mentioning the particular key-event in the code. In many cases you can use the command-case macro. It makes logical key-events easy to use and takes care of prompting and displaying help messages.

  • :abort Indicates the prompter should terminate its activity without performing any closing actions of convenience, for example.

  • :yes Indicates the prompter should take the action under consideration.

  • :no Indicates the prompter should NOT take the action under consideration.

  • :do-all Indicates the prompter should repeat the action under consideration as many times as possible.

  • :do-once Indicates the prompter should execute the action under consideration once and then exit.

  • :help Indicates the prompter should display some help information.

  • :confirm Indicates the prompter should take any input provided or use the default if the user entered nothing.

  • :quote Indicates the prompter should take the following key-event as itself without any sort of command interpretation.

  • :keep Indicates the prompter should preserve something.

  • :y Indicates a short positive response

  • :n Indicates a short negative response

Define a new logical key-event whenever:

  1. The key-event concerned represents a general class of actions, and several commands may want to take a similar action of this type.

  1. The exact key-event a command implementor chooses may generate violent taste disputes among users, and then the users can trivially change the command in their init files.

  1. You are using command-case which prevents implementors from specifying non-standard characters for dispatching in otherwise possibly portable code, and you can define and set the logical key-event in a site dependent file where you can mention implementation dependent characters.

6.12. The Echo Area

Hemlock provides a number of facilities for displaying information and prompting the user for it. Most of these work through a small area displayed at the bottom of the screen, called the Echo Area.

6.12.1. Echo Area Functions

[Function]

clear-echo-area

Description:

Clears the echo area.

[Function]

message control-string &rest format-arguments

Description:

[Function]

loud-message control-string &rest format-arguments

Description:

Displays a message in the echo area, replacing previous contents if any. loud-message is like message, but it also beeps.

[Function]

beep

Description:

Gets the user's attention, typically by making a sound.

6.12.2. Prompting Functions

Prompting functions can be used to obtain short one-line input from the user.

Cocoa note: Because of implementation restrictions, only one buffer at a time is allowed to read prompted input. If a prompting function is invoked while a prompting operation is already in effect in another buffer, the attempt fails, telling the user "Buffer xxx is already waiting for input".

Most of the prompting functions accept the following keyword arguments:

:must-exist

If :must-exist has a non-nil value then the user is prompted until a valid response is obtained. If :must-exist is nil then return as a string whatever is input. The default is t.

:default

If null input is given when the user is prompted then this value is returned. If no default is given then some input must be given before anything interesting will happen.

:default-string

If a :default is given then this is a string to be printed to indicate what the default is. The default is some representation of the value for :default, for example for a buffer it is the name of the buffer.

:prompt

This is the prompt string to display.

:help

This is similar to :prompt, except that it is displayed when the help command is typed during input.

This may also be a function. When called with no arguments, it should either return a string which is the help text or perform some action to help the user, returning nil.

[Function]

prompt-for-buffer &key :prompt :help :must-exist :default :default-string

Description:

Prompts with completion for a buffer name and returns the corresponding buffer. If must-exist is nil, then it returns the input string if it is not a buffer name. This refuses to accept the empty string as input when :default and :default-string are nil. :default-string may be used to supply a default buffer name when:default is nil, but when :must-exist is non-nil, it must name an already existing buffer.

[Function]

prompt-for-key-event &key :prompt :help

Description:

This function prompts for a key-event returning immediately when the user types the next key-event. command-case is more useful for most purposes. When appropriate, use logical key-events.

[Function]

prompt-for-key &key :prompt :help :must-exist :default :default-string

Description:

This function prompts for a key, a vector of key-events, suitable for passing to any of the functions that manipulate key bindings. If must-exist is true, then the key must be bound in the current environment, and the command currently bound is returned as the second value.

[Function]

prompt-for-file &key :prompt :help :must-exist :default :default-string

Description:

This function prompts for an acceptable filename. "Acceptable" means that it is a legal filename, and it exists if must-exist is non-nil. prompt-for-file returns a Common Lisp pathname. If the file exists as entered, then this returns it, otherwise it is merged with default as by merge-pathnames.

[Function]

prompt-for-integer &key :prompt :help :must-exist :default :default-string

Description:

This function prompts for a possibly signed integer. If must-exist is nil, then prompt-for-integer returns the input as a string if it is not a valid integer.

[Function]

prompt-for-keyword string-tables &key :prompt :help :must-exist :default :default-string

Description:

This function prompts for a keyword with completion, using the string tables in the list string-tables. If must-exist is non-nil, then the result must be an unambiguous prefix of a string in one of the string-tables, and the returns the complete string even if only a prefix of the full string was typed. In addition, this returns the value of the corresponding entry in the string table as the second value.

If must-exist is nil, then this function returns the string exactly as entered. The difference between prompt-for-keyword with must-exist nil, and prompt-for-string, is the user may complete the input using the Complete Parse and Complete Field commands.

[Function]

prompt-for-expression &key :prompt :help :must-exist :default :default-string

Description:

This function reads a Lisp expression. If must-exist is nil, and a read error occurs, then this returns the string typed.

[Function]

prompt-for-string &key :prompt :help :default :default-string

Description:

This function prompts for a string; this cannot fail.

[Function]

prompt-for-variable &key :prompt :help :must-exist :default :default-string

Description:

This function prompts for a variable name. If must-exist is non-nil, then the string must be a variable defined in the current environment, in which case the symbol name of the variable found is returned as the second value.

[Function]

prompt-for-y-or-n &key :prompt :help :must-exist :default :default-string

Description:

This prompts for logical key events :Y or :N, returning t or nil without waiting for confirmation. When the user types a confirmation key, this returns default if it is supplied. If must-exist is nil, this returns whatever key-event the user first types; however, if the user types one of the above key-events, this returns t or nil. This is analogous to the Common Lisp function y-or-n-p.

[Function]

prompt-for-yes-or-no &key :prompt :help :must-exist :default :default-string

Description:

This function is to prompt-for-y-or-n as yes-or-no-p is to y-or-n-p. "Yes" or "No" must be typed out in full and confirmation must be given.

[Macro]

command-case ({key value}*){({({tag}*) | tag} help {form}*)}*

Description:

This macro is analogous to the Common Lisp case macro. Commands such as Help use this to get a key-event, translate it to a character, and then to dispatch on the character to some case. In addition to character dispatching, this supports logical key-events by using the input key-event directly without translating it to a character. Since the description of this macro is rather complex, first consider the following example:


(defcommand "Save All Buffers" (p)
  "Give the User a chance to save each modified buffer."
  (dolist (b *buffer-list*)
    (select-buffer-command () b)
    (when (buffer-modified b)
      (command-case (:prompt "Save this buffer: [Y] "
			     :help "Save buffer, or do something else:")
	((:yes :confirm)
	 "Save this buffer and go on to the next."
	 (save-file-command () b))
	(:no "Skip saving this buffer, and go on to the next.")
	((:exit #\p) "Punt this silly loop."
	 (return nil))))))

command-case prompts for a key-event and then executes the code in the first branch with a logical key-event or a character (called tags) matching the input. Each character must be a standard-character, one that satisfies the Common Lisp standard-char-p predicate, and the dispatching mechanism compares the input key-event to any character tags by mapping the key-event to a character with ext:key-event-char. If the tag is a logical key-event, then the search for an appropriate case compares the key-event read with the tag using logical-key-event-p.

All uses of command-case have two default cases, :help and :abort. You can override these easily by specifying your own branches that include these logical key-event tags. The :help branch displays in a pop-up window the a description of the valid responses using the variously specified help strings. The :abort branch signals an editor-error.

The key/value arguments control the prompting. The following are valid values:

:help--- The default :help case displays this string in a pop-up window. In addition it formats a description of the valid input including each case's help string.

:prompt--- This is the prompt used when reading the key-event.

:bind--- This specifies a variable to which the prompting mechanism binds the input key-event. Any case may reference this variable. If you wish to know what character corresponds to the key-event, use key-event-char.

Instead of specifying a tag or list of tags, you may use t. This becomes the default branch, and its forms execute if no other branch is taken, including the default :help and :abort cases. This option has no helpstring, and the default :help case does not describe the default branch. Every command-case has a default branch; if none is specified, the macro includes one that beep's and reprompt's (see below).

Within the body of command-case, there is a defined reprompt macro. It causes the prompting mechanism and dispatching mechanism to immediately repeat without further execution in the current branch.

6.12.3. Control of Parsing Behavior

[Hemlock Variable]

Beep On Ambiguity (initial value t)

Description:

If this variable is true, then an attempt to complete a parse which is ambiguous will result in a "beep".

6.12.4. Defining New Prompting Functions

Prompting functionality is implemented by the function parse-for-something in cooperation with commands defined in "Echo Area" mode on the buffer associated with the echo area. You can implement new prompting functions by invoking parse-for-something with appropriate arguments.

[Function]

parse-for-something &key

Description:

This function enters a mode reading input from the user and echoing it in the echo area, and returns a value when done. The input is managed by commands bound in "Echo Area" mode on the buffer associated with the echo area. The following keyword arguments are accepted:

:verification-function--- This is invoked by the Confirm Parse command. It does most of the work when parsing prompted input. Confirm Parse calls it with one argument, which is the string that the user typed so far. The function should return a list of values which are to be the result of the recursive edit, or nil indicating that the parse failed. In order to return zero values, a non-nil second value may be returned along with a nil first value.

:string-tables--- This is the list of string-tables, if any, that pertain to this parse.

:value-must-exist--- This is referred to by the verification function, and possibly some of the commands.

:default--- The string representing the default object when prompting the user. Confirm Parse supplies this to the parse verification function when the user input is empty.

:default-string--- When prompting the user, if :default is not specified, Hemlock displays this string as a representation of the default object; for example, when prompting for a buffer, this argument would be a default buffer name.

:type--- The kind of parse, e.g. :file, :keyword, :string. This tells the completion commands how to do completion, with :string disabling completion.

:prompt--- The prompt to display to the user.

:help--- The help string or function being used for the current parse.

6.12.5. Some Echo Area Commands

These are some of the Echo Area commands that coordinate with the prompting routines. Hemlock binds other commands specific to the Echo Area, but they are uninteresting to mention here, such as deleting to the beginning of the line or deleting backwards a word.

[Command]

Help On Parse (bound to Home, C-_ in Echo Area mode)

Description:

Display the help text for the parse currently in progress.

[Command]

Complete Keyword (bound to Escape in Echo Area mode)

Description:

This attempts to complete the current region. It signals an editor-error if the input is ambiguous or incorrect.

[Command]

Complete Field (bound to Space in Echo Area mode)

Description:

Similar to Complete Keyword, but only attempts to complete up to and including the first character in the keyword with a non-zero :parse-field-separator attribute. If there is no field separator then attempt to complete the entire keyword. If it is not a keyword parse then just self-insert.

[Command]

Confirm Parse (bound to Return in Echo Area mode)

Description:

Call the verification function with the current input. If it returns a non-nil value then that is returned as the value of the parse. A parse may return a nil value if the verification function returns a non-nil second value.

6.13. Files

This chapter discusses ways to read and write files at various levels---at marks, into regions, and into buffers. This also treats automatic mechanisms that affect the state of buffers in which files are read.

6.13.1. File Options and Type Hooks

The user specifies file options with a special syntax on the first line of a file. If the first line contains the string "-*-", then Hemlock interprets the text between the first such occurrence and the second, which must be contained in one line , as a list of "option: value" pairs separated by semicolons. The following is a typical example:


;;; -*- Mode: Lisp, Editor; Package: Hemlock -*-

See the Hemlock User's Manual for more details and predefined options.

File type hooks are executed when Hemlock reads a file into a buffer based on the type of the pathname. When the user specifies a Mode file option that turns on a major mode, Hemlock ignores type hooks. This mechanism is mostly used as a simple means for turning on some appropriate default major mode.

[Macro]

define-file-option name (buffer value) {declaration}* {form}*

Description:

This defines a new file option with the string name name. Buffer and value specify variable names for the buffer and the option value string, and forms are evaluated with these bound.

[Macro]

define-file-type-hook type-list (buffer type) {declaration}* {form}*

Description:

This defines some code that process-file-options(below) executes when the file options fail to set a major mode. This associates each type, a string, in type-list with a routine that binds buffer to the buffer the file is in and type to the type of the pathname.

[Function]

process-file-options buffer &optional pathname

Description:

This checks for file options in buffer and invokes handlers if there are any. Pathname defaults to buffer's pathname but may be nil. If there is no Mode file option that specifies a major mode, and pathname has a type, then this tries to invoke the appropriate file type hook. read-buffer-file calls this.

6.13.2. Pathnames and Buffers

There is no good way to uniquely identify buffer names and pathnames. However, Hemlock has one way of mapping pathnames to buffer names that should be used for consistency among customizations and primitives. Independent of this, Hemlock provides a means for consistently generating prompting defaults when asking the user for pathnames.

[Function]

pathname-to-buffer-name pathname

Description:

This function returns a string of the form "file-namestring directory-namestring".

[Hemlock Variable]

Pathname Defaults (initial value (pathname "gazonk.del"))

Description:

[Hemlock Variable]

Last Resort Pathname Defaults Function

Description:

[Hemlock Variable]

Last Resort Pathname Defaults (initial value (pathname "gazonk"))

Description:

These variables control the computation of default pathnames when needed for promting the user. Pathname Defaults is a sticky default. See the Hemlock User's Manual for more details.

[Function]

buffer-default-pathname buffer

Description:

This returns Buffer Pathname if it is bound. If it is not bound, and buffer's name is composed solely of alphnumeric characters, then return a pathname formed from buffer's name. If buffer's name has other characters in it, then return the value of Last Resort Pathname Defaults Function called on buffer.

6.13.3. File Groups

Currently Hemlock doesn't have support for file groups.

6.13.4. File Reading and Writing

Common Lisp pathnames are used by the file primitives. For probing, checking write dates, and so forth, all of the Common Lisp file functions are available.

[Function]

read-file pathname mark

Description:

This inserts the file named by pathname at mark.

[Function]

write-file region pathname &key :keep-backup :append

Description:

[Hemlock Variable]

Keep Backup Files (initial value nil)

Description:

This function writes the contents of region to the file named by pathname. This writes region using a stream as if it were opened with :if-exists supplied as :rename-and-delete.

When keep-backup, which defaults to the value of Keep Backup Files, is non-nil, this opens the stream as if :if-exists were :rename. If append is non-nil, this writes the file as if it were opened with:if-exists supplied as :append.

This signals an error if both append and keep-backup are supplied as non-nil.

[Function]

write-buffer-file buffer pathname

Description:

[Hemlock Variable]

Write File Hook

Description:

[Hemlock Variable]

Add Newline at EOF on Writing File (initial value :ask-user)

Description:

write-buffer-file writes buffer to the file named by pathname including the following:

  • It assumes pathname is somehow related to buffer's pathname: if the buffer's write date is not the same as pathname's, then this prompts the user for confirmation before overwriting the file.

  • It consults Add Newline at EOF on Writing File (see Hemlock User's Manual for possible values) and interacts with the user if necessary.

  • It sets Pathname Defaults, and after using write-file, marks buffer unmodified.

  • It updates Buffer's pathname and write date.

  • It renames the buffer according to the new pathname if possible.

  • It invokes Write File Hook.

Write File Hook is a list of functions that take the newly written buffer as an argument.

[Function]

read-buffer-file pathname buffer

Description:

[Hemlock Variable]

Read File Hook

Description:

read-buffer-file deletes buffer's region and uses read-file to read pathname into it, including the following:

  • It sets buffer's write date to the file's write date if the file exists; otherwise, it messages that this is a new file and sets buffer's write date to nil.

  • It moves buffer's point to the beginning.

  • It sets buffer's unmodified status.

  • It sets buffer's pathname to the result of probing pathname if the file exists; otherwise, this function sets buffer's pathname to the result of merging pathname with default-directory.

  • It sets Pathname Defaults to the result of the previous item.

  • It processes the file options.

  • It invokes Read File Hook.

Read File Hook is a list functions that take two arguments---the buffer read into and whether the file existed, t if so.

6.14. Hemlock's Lisp Environment

This chapter is sort of a catch all for any functions and variables which concern Hemlock's interaction with the outside world.

6.14.1. Entering and Leaving the Editor

[Function]

ed &optional x

Description:

This a standard Common Lisp function. If x is supplied and is a string or pathname, the file specified by x is visited in a hemlock view (opening a new window if necessary, otherwise bringing an existing window with the file to the front), and the hemlock view object is the return value from the function.

If x is null, a new empty hemlock view is created and returned.

If x is a symbol or a setf function name, it attempts to edit the definition of the name. In this last case, the function returns without waiting for the operation to complete (for example, it might put up a non-modal dialog asking the user to select one of multiple definitions) and hence the return value is always NIL.

6.14.2. Keyboard Input

[Variable]

*key-event-history*

Description:

This is a Hemlock ring buffer that holds the last 60 key-events received.

[Function]

last-key-event-typed

Description:

This function returns the last key-event the user typed to invoke the current command.

[Function]

last-char-typed

Description:

This function returns the character corresponding to the last key event typed.

6.14.3. Hemlock Streams

It is possible to create streams which output to or get input from a buffer. This mechanism is quite powerful and permits easy interfacing of Hemlock to Lisp.

Note that operations on these streams operate directly on buffers, therefore they have the same restrictions as described here for interacting with buffers from outside of the GUI thread.

[Function]

make-hemlock-output-stream mark &optional buffered

Description:

This function returns a stream that inserts at mark all output directed to it. It works best if mark is a left-inserting mark. Buffered controls whether the stream is buffered or not, and its valid values are the following keywords:

:none--- No buffering is done. This is the default.

:line--- The buffer is flushed whenever a newline is written or when it is explicitly done with force-output.

:full--- The stream is only brought up to date when it is explicitly done with force-output

[Function]

hemlock-output-stream-p object

Description:

This function returns t if object is a hemlock-output-stream object.

[Function]

make-hemlock-region-stream region

Description:

This function returns a stream from which the text in region can be read.

[Function]

hemlock-region-stream-p object

Description:

This function returns t if object is a hemlock-region-stream object.

[Macro]

with-input-from-region (var region) {declaration}* {form}*

Description:

While evaluating forms, binds var to a stream which returns input from region.

[Macro]

with-output-to-mark (var mark [buffered]) {declaration}* {form}*

Description:

During the evaluation of the forms, binds var to a stream which inserts output at the permanent mark. Buffered has the same meaning as for make-hemlock-output-stream.

[Macro]

with-pop-up-display (var &key height name) {declaration}* {form}*

Description:

This macro executes forms in a context with var bound to a stream. Hemlock collects output to this stream and tries to pop up a display of the appropriate height containing all the output. When height is supplied, Hemlock creates the pop-up display immediately, forcing output on line breaks. This is useful for displaying information of temporary interest.

6.14.4. Interface to the Error System

Hemlock commands are executed from an event handler in the initial Cocoa thread. They are executed within a ccl::with-standard-abort-handling form, which means cl:abort, ccl:abort-break, ccl:throw-cancel will abort the current command only and exit the event handler in an orderly fashion.

In addition, for now, lisp errors during command execution dump a backtrace in the system console and are otherwise handled as if by handle-lisp-errors below, which means it is not possible to debug errors at the point of the error. Once Clozure CL has better support for debugging errors in the initial Cocoa thread, better Hemlock error handling will be provided that will allow for some way to debug.

[Function]

editor-error &rest args

Description:

This function is called to report minor errors to the user. These are errors that a normal user could encounter in the course of editing, such as a search failing or an attempt to delete past the end of the buffer. This function simply aborts the current command. Any args specified are used to format an error message to be placed in the echo area. This function never returns.

[Macro]

handle-lisp-errors {form}*

Description:

Within the body of this macro any Lisp errors that occur are handled by displaying an error message in a dialog and aborting the current command, leaving the error text in the echo area. This macro should be wrapped around code which may get an error due to some action of the user --- for example, evaluating code fragments on the behalf of and supplied by the user.

6.14.5. Definition Editing

Hemlock provides commands for finding the definition of a function or variable and placing the user at the definition in a buffer. A function is provided to allow invoking this functionality outside of Hemlock. Note that this function is unusual in that it is it is safe to call outside of the command interpreter, and in fact it can be called from any thread.

[Function]

edit-definition name

Description:

This function tries to find the definition of name, create or activate the window containing it, and scroll the view to show the definition. If there are multiple definitions available, the user is given a choice of which one to use. This function may return before the operation is complete.

6.14.6. Event Scheduling

No Event Scheduling functionality is provided at this time.

6.14.7. Miscellaneous

[Function]

in-lisp {form}*

Description:

This evaluates forms inside handle-lisp-errors. It also binds *package* to the package named by Current Package if it is non-nil. Use this when evaluating Lisp code on behalf of the user.

[Macro]

do-alpha-chars (var kind [result]) {form}*

Description:

This iterates over alphabetic characters in Common Lisp binding var to each character in order as specified under character relations in Common Lisp the Language. Kind is one of:lower, :upper, or :both. When the user supplies :both, lowercase characters are processed first.

6.15. High-Level Text Primitives

This chapter discusses primitives that operate on higher level text forms than characters and words. For English text, there are functions that know about sentence and paragraph structures, and for Lisp sources, there are functions that understand this language. This chapter also describes mechanisms for organizing file sections into logical pages and for formatting text forms.

6.15.1. Indenting Text

[Hemlock Variable]

Indent Function (initial value tab-to-tab-stop)

Description:

The value of this variable determines how indentation is done, and it is a function which is passed a mark as its argument. The function should indent the line that the mark points to. The function may move the mark around on the line. The mark will be :left-inserting. The default simply inserts a tab character at the mark. A function for Lisp mode probably moves the mark to the beginning of the line, deletes horizontal whitespace, and computes some appropriate indentation for Lisp code.

[Hemlock Variable]

Indent with Tabs (initial value nil)

Description:

[Hemlock Variable]

Spaces per Tab (initial value 8)

Description:

Indent with Tabs should be true if indenting should use tabs whenever possible. If nil, the default, it only uses spaces. Spaces per Tab defines the size of a tab.

[Function]

indent-region region

Description:

[Function]

indent-region-for-commands region

Description:

indent-region invokes the value of Indent Function on every line of region. indent-region-for-commands uses indent-region but first saves the region for the Undo command.

[Function]

delete-horizontal-space mark

Description:

This deletes all characters on either side of mark with a Space attribute (see System Defined Character Attributes) of 1.

6.15.2. Lisp Text Buffers

Hemlock bases its Lisp primitives on parsing a block of the buffer and annotating lines as to what kind of Lisp syntax occurs on the line or what kind of form a mark might be in (for example, string, comment, list, etc.). These do not work well if the block of parsed forms is exceeded when moving marks around these forms, but the block that gets parsed is somewhat programmable.

There is also a notion of a top level form which this documentation often uses synonymously with defun, meaning a Lisp form occurring in a source file delimited by parentheses with the opening parenthesis at the beginning of some line. The names of the functions include this inconsistency.

[Function]

pre-command-parse-check mark for-sure

Description:

[Hemlock Variable]

Parse Start Function (initial value start-of-parse-block)

Description:

[Hemlock Variable]

Parse End Function (initial value end-of-parse-block)

Description:

[Hemlock Variable]

Minimum Lines Parsed (initial value 50)

Description:

[Hemlock Variable]

Maximum Lines Parsed (initial value 500)

Description:

[Hemlock Variable]

Defun Parse Goal (initial value 2)

Description:

pre-command-parse-check calls Parse Start Function and Parse End Function on mark to get two marks. It then parses all the lines between the marks including the complete lines they point into. When for-sure is non-nil, this parses the area regardless of any cached information about the lines. Every command that uses the following routines calls this before doing so.

The default values of the start and end variables use Minimum Lines Parsed, Maximum Lines Parsed, and Defun Parse Goal to determine how big a region to parse. These two functions always include at least the minimum number of lines before and after the mark passed to them. They try to include Defun Parse Goal number of top level forms before and after the mark passed them, but these functions never return marks that include more than the maximum number of lines before or after the mark passed to them.

[Function]

form-offset mark count

Description:

This tries to move mark count forms forward if positive or -count forms backwards if negative. Mark is always moved. If there were enough forms in the appropriate direction, this returns mark, otherwise nil.

[Function]

top-level-offset mark count

Description:

This tries to move mark count top level forms forward if positive or -count top level forms backwards if negative. If there were enough top level forms in the appropriate direction, this returns mark, otherwise nil. Mark is moved only if this is successful.

[Function]

mark-top-level-form mark1 mark2

Description:

This moves mark1 and mark2 to the beginning and end, respectively, of the current or next top level form. Mark1 is used as a reference to start looking. The marks may be altered even if unsuccessful. If successful, return mark2, else nil. Mark2 is left at the beginning of the line following the top level form if possible, but if the last line has text after the closing parenthesis, this leaves the mark immediately after the form.

[Function]

defun-region mark

Description:

This returns a region around the current or next defun with respect to mark. Mark is not used to form the region. If there is no appropriate top level form, this signals an editor-error. This calls pre-command-parse-check first.

[Function]

inside-defun-p mark

Description:

[Function]

start-defun-p mark

Description:

These return, respectively, whether mark is inside a top level form or at the beginning of a line immediately before a character whose Lisp Syntax (see System Defined Character Attributes) value is :opening-paren.

[Function]

forward-up-list mark

Description:

[Function]

backward-up-list mark

Description:

Respectively, these move mark immediately past a character whose Lisp Syntax (see System Defined Character Attributes) value is :closing-paren or immediately before a character whose Lisp Syntax value is :opening-paren.

[Function]

valid-spot mark forwardp

Description:

This returns t or nil depending on whether the character indicated by mark is a valid spot. When forwardp is set, use the character after mark and vice versa. Valid spots exclude commented text, inside strings, and character quoting.

[Function]

defindent name count

Description:

This defines the function with name to have count special arguments. indent-for-lisp, the value of Indent Function in Lisp mode, uses this to specially indent these arguments. For example, do has two, with-open-file has one, etc. There are many of these defined by the system including definitions for special Hemlock forms. Name is a simple-string, case insensitive and purely textual (that is, not read by the Lisp reader); therefore, "with-a-mumble" is distinct from "mumble:with-a-mumble".

6.15.3. English Text Buffers

This section describes some routines that understand basic English language forms.

[Function]

word-offset mark count

Description:

This moves mark count words forward (if positive) or backwards (if negative). If mark is in the middle of a word, that counts as one. If there were count (-count if negative) words in the appropriate direction, this returns mark, otherwise nil. This always moves mark. A word lies between two characters whose Word Delimiter attribute value is 1 (see System Defined Character Attributes).

[Function]

sentence-offset mark count

Description:

This moves mark count sentences forward (if positive) or backwards (if negative). If mark is in the middle of a sentence, that counts as one. If there were count (-count if negative) sentences in the appropriate direction, this returns mark, otherwise nil. This always moves mark.

A sentence ends with a character whose Sentence Terminator attribute is 1 followed by two spaces, a newline, or the end of the buffer. The terminating character is optionally followed by any number of characters whose Sentence Closing Char attribute is 1. A sentence begins after a previous sentence ends, at the beginning of a paragraph, or at the beginning of the buffer.

[Function]

paragraph-offset mark count &optional prefix

Description:

[Hemlock Variable]

Paragraph Delimiter Function (initial value )

Description:

This moves mark count paragraphs forward (if positive) or backwards (if negative). If mark is in the middle of a paragraph, that counts as one. If there were count (-count if negative) paragraphs in the appropriate direction, this returns mark, otherwise nil. This only moves mark if there were enough paragraphs.

Paragraph Delimiter Function holds a function that takes a mark, typically at the beginning of a line, and returns whether or not the current line should break the paragraph. default-para-delim-function returns t if the next character, the first on the line, has a Paragraph Delimiter attribute value of 1. This is typically a space, for an indented paragraph, or a newline, for a block style. Some modes require a more complicated determinant; for example, Scribe modes adds some characters to the set and special cases certain formatting commands.

Prefix defaults to Fill Prefix, and the right prefix is necessary to correctly skip paragraphs. If prefix is non-nil, and a line begins with prefix, then the scanning process skips the prefix before invoking the Paragraph Delimiter Function. Note, when scanning for paragraph bounds, and prefix is non-nil, lines are potentially part of the paragraph regardless of whether they contain the prefix; only the result of invoking the delimiter function matters.

The programmer should be aware of an idiom for finding the end of the current paragraph. Assume paragraphp is the result of moving mark one paragraph, then the following correctly determines whether there actually is a current paragraph:


(or paragraphp
  (and (last-line-p mark)
       (end-line-p mark)
       (not (blank-line-p (mark-line mark))))) 

In this example mark is at the end of the last paragraph in the buffer, and there is no last newline character in the buffer. paragraph-offset would have returned nil since it could not skip any paragraphs since mark was at the end of the current and last paragraph. However, you still have found a current paragraph on which to operate. mark-paragraph understands this problem.

[Function]

mark-paragraph mark1 mark2

Description:

This marks the next or current paragraph, setting mark1 to the beginning and mark2 to the end. This uses Fill Prefix. Mark1 is always on the first line of the paragraph, regardless of whether the previous line is blank. Mark2 is typically at the beginning of the line after the line the paragraph ends on, this returns mark2 on success. If this cannot find a paragraph, then the marks are left unmoved, and nil is returned.

6.15.4. Logical Pages

Logical pages are not supported at this time.

6.15.5. Filling

Filling is an operation on text that breaks long lines at word boundaries before a given column and merges shorter lines together in an attempt to make each line roughly the specified length. This is different from justification which tries to add whitespace in awkward places to make each line exactly the same length. Hemlock's filling optionally inserts a specified string at the beginning of each line. Also, it eliminates extra whitespace between lines and words, but it knows two spaces follow sentences.

[Hemlock Variable]

Fill Column (initial value 75)

Description:

[Hemlock Variable]

Fill Prefix (initial value nil)

Description:

These variables hold the default values of the prefix and column arguments to Hemlock's filling primitives. If Fill Prefix is nil, then there is no fill prefix.

[Function]

fill-region region &optional prefix column

Description:

This deletes any blank lines in region and fills it according to prefix and column. Prefix and column default to Fill Prefix and Fill Column.

[Function]

fill-region-by-paragraphs region &optional prefix column

Description:

This finds paragraphs within region and fills them with fill-region. This ignores blank lines between paragraphs. Prefix and column default to Fill Prefix and Fill Column.

6.16. Utilities

This chapter describes a number of utilities for manipulating some types of objects Hemlock uses to record information. String-tables are used to store names of variables, commands, modes, and buffers. Ring lists can be used to provide a kill ring, recent command history, or other user-visible features.

6.16.1. String-table Functions

String tables are similar to Common Lisp hash tables in that they associate a value with an object. There are a few useful differences: in a string table the key is always a case insensitive string, and primitives are provided to facilitate keyword completion and recognition. Any type of string may be added to a string table, but the string table functions always return simple-string's.

A string entry in one of these tables may be thought of as being separated into fields or keywords. The interface provides keyword completion and recognition which is primarily used to implement some Echo Area commands. These routines perform a prefix match on a field-by-field basis allowing the ambiguous specification of earlier fields while going on to enter later fields. While string tables may use any string-char as a separator, the use of characters other than space may make the Echo Area commands fail or work unexpectedly.

[Function]

make-string-table

Description:

This function creates an empty string table that uses separator as the character, which must be a string-char, that distinguishes fields. Initial-contents specifies an initial set of strings and their values in the form of a dotted a-list, for example:


'(("Global" . t) ("Mode" . t) ("Buffer" . t))

[Function]

string-table-p string-table

Description:

This function returns t if string-table is a string-table object, otherwise nil.

[Function]

string-table-separator string-table

Description:

This function returns the separator character given to make-string-table.

[Function]

delete-string string table

Description:

[Function]

clrstring table

Description:

delete-string removes any entry for string from the string-table table, returning t if there was an entry. clrstring removes all entries from table.

[Function]

getstring string table

Description:

This function returns as multiple values, first the value corresponding to the string if it is found and nil if it isn't, and second t if it is found and nil if it isn't.

This may be set with setf to add a new entry or to store a new value for a string. It is an error to try to insert a string with more than one field separator character occurring contiguously.

[Function]

complete-string string tables

Description:

This function completes string as far as possible over the list of tables, returning five values. It is an error for tables to have different separator characters. The five return values are as follows:

  • The maximal completion of the string or nil if there is none.

  • An indication of the usefulness of the returned string:

:none--- There is no completion of string.

:complete--- The completion is a valid entry, but other valid completions exist too. This occurs when the supplied string is an entry as well as initial substring of another entry.

:unique--- The completion is a valid entry and unique.

:ambiguous--- The completion is invalid; get-string would return nil and nil if given the returned string.

  • The value of the string when the completion is :unique or :complete, otherwise nil.

  • An index, or nil, into the completion returned, indicating where the addition of a single field to string ends. The command Complete Field uses this when the completion contains the addition to string of more than one field.

  • An index to the separator following the first ambiguous field when the completion is :ambiguous or :complete, otherwise nil.

[Function]

find-ambiguous string table

Description:

[Function]

find-containing string table

Description:

find-ambiguous returns a list in alphabetical order of all the strings in table matching string. This considers an entry as matching if each field in string, taken in order, is an initial substring of the entry's fields; entry may have fields remaining.

find-containing is similar, but it ignores the order of the fields in string, returning all strings in table matching any permutation of the fields in string.

[Macro]

do-strings (string-var value-var table result) declaration tag statement

Description:

This macro iterates over the strings in table in alphabetical order. On each iteration, it binds string-var to an entry's string and value-var to an entry's value.

6.16.2. Ring Functions

There are various purposes in an editor for which a ring of values can be used, so Hemlock provides a general ring buffer type. It is used for maintaining a ring of killed regions, a ring of marks, or a ring of command strings which various modes and commands maintain as a history mechanism.

[Function]

make-ring length &optional delete-function

Description:

Makes an empty ring object capable of holding up to length Lisp objects. Delete-function is a function that each object is passed to before it falls off the end. Length must be greater than zero.

[Function]

ringp ring

Description:

Returns t if ring is a ring object, otherwise nil.

[Function]

ring-length ring

Description:

Returns as multiple-values the number of elements which ring currently holds and the maximum number of elements which it may hold.

[Function]

ring-ref ring index

Description:

Returns the index'th item in the ring, where zero is the index of the most recently pushed. This may be set with setf.

[Function]

ring-push object ring

Description:

Pushes object into ring, possibly causing the oldest item to go away.

[Function]

ring-pop ring

Description:

Removes the most recently pushed object from ring and returns it. If the ring contains no elements then an error is signalled.

[Function]

rotate-ring ring offset

Description:

With a positive offset, rotates ring forward that many times. In a forward rotation the index of each element is reduced by one, except the one which initially had a zero index, which is made the last element. A negative offset rotates the ring the other way.

6.16.3. Undoing commands

No API to the undo facility is provided at this time.

6.17. Miscellaneous

This chapter is somewhat of a catch-all for comments and features that don't fit well anywhere else.

6.17.1. Key-events

6.17.1.1. Introduction

The canonical representation of editor input is a key-event structure. Users can bind commands to keys, which are non-empty sequences of key-events. A key-event consists of an identifying token known as a keysym and a field of bits representing modifiers. Users define keysyms by supplying names that reflect the legends on their keyboard's keys. Users define modifier names similarly, but the system chooses the bit and mask for recognizing the modifier. You can use keysym and modifier names to textually specify key-events and Hemlock keys in a #k syntax. The following are some examples:


#k"C-u"
#k"Control-u"
#k"c-m-z"
#k"control-x meta-d"
#k"a"
#k"A"
#k"Linefeed"

This is convenient for use within code and in init files containing bind-key calls.

The #k syntax is delimited by double quotes, but the system parses the contents rather than reading it as a Common Lisp string. Within the double quotes, spaces separate multiple key-events. A single key-event optionally starts with modifier names terminated by hyphens. Modifier names are alphabetic sequences of characters which the system uses case-insensitively. Following modifiers is a keysym name, which is case-insensitive if it consists of multiple characters, but if the name consists of only a single character, then it is case-sensitive.

You can escape special characters --- hyphen, double quote, open angle bracket, close angle bracket, and space --- with a backslash, and you can specify a backslash by using two contiguously. You can use angle brackets to enclose a keysym name with many special characters in it. Between angle brackets appearing in a keysym name position, there are only two special characters, the closing angle bracket and backslash.

6.17.1.2. Interface

[Function]

define-keysym keysym preferred-name &rest other-names

Description:

This function establishes a mapping from preferred-name to keysym for purposes of #k syntax. Other-names also map to keysym, but the system uses preferred-name when printing key-events. The names are case-insensitive simple-strings; however, if the string contains a single character, then it is used case-sensitively. Redefining a keysym or re-using names has undefined effects.

Keysym can be any object, but generally it is either an integer representing the window-system code for the event, or a keyword which allows the mapping of the keysym to its code to be defined separately.

[Function]

define-keysym-code keysym code

Description:

Defines the window-system code for the key event which in Hemlock is represented by keysym.

[Function]

define-mouse-keysym button keysym name shifted-bit event-key

Description:

This function defines keysym named name for key-events representing mouse click events. Shifted-bit is a defined modifier name that translate-mouse-key-event sets in the key-event it returns whenever the shift bit is set in an incoming event.

[Function]

name-keysym name

Description:

This function returns the keysym named name. If name is unknown, this returns nil.

[Function]

keysym-names keysym

Description:

This function returns the list of all names for keysym. If keysym is undefined, this returns nil.

[Function]

keysym-preferred-name keysym

Description:

This returns the preferred name for keysym, how it is typically printed. If keysym is undefined, this returns nil.

[Function]

define-key-event-modifier long-name short-name

Description:

This establishes long-name and short-name as modifier names for purposes of specifying key-events in #k syntax. The names are case-insensitive strings. If either name is already defined, this signals an error.

The system defines the following default modifiers (first the long name, then the short name):

  • "Hyper", "H"

  • "Super", "S"

  • "Meta", "M"

  • "Control", "C"

  • "Shift", "Shift"

  • "Lock", "Lock"

[Variable]

*all-modifier-names*

Description:

This variable holds all the defined modifier names.

[Function]

make-key-event-bits &rest modifier-names

Description:

This function returns bits suitable for make-key-event from the supplied modifier-names. If any name is undefined, this signals an error.

[Function]

key-event-modifier-mask modifier-name

Description:

This function returns a mask for modifier-name. This mask is suitable for use with key-event-bits. If modifier-name is undefined, this signals an error.

[Function]

key-event-bits-modifiers bits

Description:

This returns a list of key-event modifier names, one for each modifier set in bits.

[Function]

make-key-event object bits

Description:

This function returns a key-event described by object with bits. Object is one of keysym, string, or key-event. When object is a key-event, this uses key-event-keysym. You can form bits with make-key-event-bits or key-event-modifier-mask.

[Function]

key-event-p object

Description:

This function returns whether object is a key-event.

[Function]

key-event-bits key-event

Description:

This function returns the bits field of a key-event.

[Function]

key-event-keysym key-event

Description:

This function returns the keysym field of a key-event.

[Function]

char-key-event character

Description:

This function returns the key-event associated with character. You can associate a key-event with a character by setf-ing this form.

[Function]

key-event-char key-event

Description:

This function returns the character associated with key-event. You can associate a character with a key-event by setf'ing this form. The system defaultly translates key-events in some implementation dependent way for text insertion; for example, under an ASCII system, the key-event #k"C-h", as well as #k"backspace" would map to the Common Lisp character that causes a backspace.

[Function]

key-event-bit-p key-event bit-name

Description:

This function returns whether key-event has the bit set named by bit-name. This signals an error if bit-name is undefined.

[Macro]

do-alpha-key-events (var kind &optional result) form

Description:

This macro evaluates each form with var bound to a key-event representing an alphabetic character. Kind is one of :lower, :upper, or :both, and this binds var to each key-event in order a-z A-Z. When :both is specified, this processes lowercase letters first.

[Function]

pretty-key-string key &optional stream long-names-p

Description:

This returns a string representing key, a key-event or vector of key-events, in a user-expected fashion. Long-names-p indicates whether modifiers should be described by their long or short name.

Chapter 7. Programming with Threads

7.1. Threads Overview

Clozure CL provides facilities which enable multiple threads of execution (threads, sometimes called lightweight processes or just processes, though the latter term shouldn't be confused with the OS's notion of a process) within a lisp session. This document describes those facilities and issues related to multithreaded programming in Clozure CL.

Wherever possible, I'll try to use the term "thread" to denote a lisp thread, even though many of the functions in the API have the word "process" in their name. A lisp-process is a lisp object (of type CCL:PROCESS) which is used to control and communicate with an underlying native thread. Sometimes, the distinction between these two (quite different) objects can be blurred; other times, it's important to maintain.

Lisp threads share the same address space, but maintain their own execution context (stacks and registers) and their own dynamic binding context.

Traditionally, Clozure CL's threads have been cooperatively scheduled: through a combination of compiler and runtime support, the currently executing lisp thread arranged to be interrupted at certain discrete points in its execution (typically on entry to a function and at the beginning of any looping construct). This interrupt occurred several dozen times per second; in response, a handler function might observe that the current thread had used up its time slice and another function (the lisp scheduler) would be called to find some other thread that was in a runnable state, suspend execution of the current thread, and resume execution of the newly executed thread. The process of switching contexts between the outgoing and incoming threads happened in some mixture of Lisp and assembly language code; as far as the OS was concerned, there was one native thread running in the Lisp image and its stack pointer and other registers just happened to change from time to time.

Under Clozure CL's cooperative scheduling model, it was possible (via the use of the CCL:WITHOUT-INTERRUPTS construct) to defer handling of the periodic interrupt that invoked the lisp scheduler; it was not uncommon to use WITHOUT-INTERRUPTS to gain safe, exclusive access to global data structures. In some code (including much of Clozure CL itself) this idiom was very common: it was (justifiably) believed to be an efficient way of inhibiting the execution of other threads for a short period of time.

The timer interrupt that drove the cooperative scheduler was only able to (pseudo-)preempt lisp code: if any thread called a blocking OS I/O function, no other thread could be scheduled until that thread resumed execution of lisp code. Lisp library functions were generally attuned to this constraint, and did a complicated mixture of polling and "timed blocking" in an attempt to work around it. Needless to say, this code is complicated and less efficient than it might be; it meant that the lisp was a little busier than it should have been when it was "doing nothing" (waiting for I/O to be possible.)

For a variety of reasons - better utilization of CPU resources on single and multiprocessor systems and better integration with the OS in general - threads in Clozure CL 0.14 and later are preemptively scheduled. In this model, lisp threads are native threads and all scheduling decisions involving them are made by the OS kernel. (Those decisions might involve scheduling multiple lisp threads simultaneously on multiple processors on SMP systems.) This change has a number of subtle effects:

  • it is possible for two (or more) lisp threads to be executing simultaneously, possibly trying to access and/or modify the same data structures. Such access really should have been coordinated through the use of synchronization objects regardless of the scheduling modeling effect; preemptively scheduled threads increase the chance of things going wrong at the wrong time and do not offer lightweight alternatives to the use of those synchronization objects.

  • even on a single-processor system, a context switch can happen on any instruction boundary. Since (in general) other threads might allocate memory, this means that a GC can effectively take place at any instruction boundary. That's mostly an issue for the compiler and runtime system to be aware of, but it means that certain practices(such as trying to pass the address of a lisp object to foreign code)that were always discouraged are now discouraged ... vehemently.

  • there is no simple and efficient way to "inhibit the scheduler"or otherwise gain exclusive access to the entire CPU.

  • There are a variety of simple and efficient ways to synchronize access to particular data structures.

As a broad generalization: code that's been aggressively tuned to the constraints of the cooperative scheduler may need to be redesigned to work well with the preemptive scheduler (and code written to run under Clozure CL's interface to the native scheduler may be less portable to other CL implementations, many of which offer a cooperative scheduler and an API similar to Clozure CL (< 0.14) 's.) At the same time, there's a large overlap in functionality in the two scheduling models, and it'll hopefully be possible to write interesting and useful MP code that's largely independent of the underlying scheduling details.

The keyword :OPENMCL-NATIVE-THREADS is on *FEATURES* in 0.14 and later and can be used for conditionalization where required.

7.2. (Intentionally) Missing Functionality

Much of the functionality described above is similar to that provided by Clozure CL's cooperative scheduler, some other parts of which make no sense in a native threads implementation.

  • PROCESS-RUN-REASONS and PROCESS-ARREST-REASONS were SETFable process attributes; each was just a list of arbitrary tokens. A thread was eligible for scheduling (roughly equivalent to being "enabled") if its arrest-reasons list was empty and its run-reasons list was not. I don't think that it's appropriate to encourage a programming style in which otherwise runnable threads are enabled and disabled on a regular basis (it's preferable for threads to wait for some sort of synchronization event to occur if they can't occupy their time productively.)

  • There were a number of primitives for maintaining process queues;that's now the OS's job.

  • Cooperative threads were based on coroutining primitives associated with objects of type STACK-GROUP. STACK-GROUPs no longerexist.

7.3. Implementation Decisions and Open Questions

7.3.1. Thread Stack Sizes

When you use MAKE-PROCESS to create a thread, you can specify a stack size. Clozure CL does not impose a limit on the stack size you choose, but there is some evidence that choosing a stack size larger than the operating system's limit can cause excessive paging activity, at least on some operating systems.

The maximum stack size is operating-system-dependent. You can use shell commands to determine what it is on your platform. In bash, use "ulimit -s -H" to find the limit; in tcsh, use "limit -h s".

This issue does not affect programs that create threads using the default stack size, which you can do either by specifying no value for the :stack-size argument to MAKE-PROCESS, or by specifying the value CCL::*default-control-stack-size*.

If your program creates threads with a specified stack size, and that size is larger than the OS-specified limit, you may want to consider reducing the stack size in order to avoid possible excessive paging activity.

7.3.2.  As of August 2003:

  • It's not clear that exposing PROCESS-SUSPEND/PROCESS-RESUME is a good idea: it's not clear that they offer ways to win, and it's clear that they offer ways to lose.

  • It has traditionally been possible to reset and enable a process that's "exhausted" . (As used here, the term "exhausted" means that the process's initial function has run and returned and the underlying native thread has been deallocated.) One of the principal uses of PROCESS-RESET is to "recycle" threads; enabling an exhausted process involves creating a new native thread (and stacks and synchronization objects and ...),and this is the sort of overhead that such a recycling scheme is seeking to avoid. It might be worth trying to tighten things up and declare that it's an error to apply PROCESS-ENABLE to an exhausted thread (and to make PROCESS-ENABLE detect this error.)

  • When native threads that aren't created by Clozure CL first call into lisp, a "foreign process" is created, and that process is given its own set of initial bindings and set up to look mostly like a process that had been created by MAKE-PROCESS. The life cycle of a foreign process is certainly different from that of a lisp-created one: it doesn't make sense to reset/preset/enable a foreign process, and attempts to perform these operations should be detected and treated as errors.

7.4. Porting Code from the Old Thread Model

Older versions of Clozure CL used what are often called "user-mode threads", a less versatile threading model which does not require specific support from the operating system. This section discusses how to port code which was written for that mode.

It's hard to give step-by-step instructions; there are certainly a few things that one should look at carefully:

  • It's wise to be suspicious of most uses of WITHOUT-INTERRUPTS; there may be exceptions, but WITHOUT-INTERRUPTS is often used as shorthand for WITH-APPROPRIATE-LOCKING. Determining what type of locking is appropriate and writing the code to implement it is likely to be straightforward and simple most of the time.

  • I've only seen one case where a process's "run reasons" were used to communicate information as well as to control execution; I don't think that this is a common idiom, but may be mistaken about that.

  • It's certainly possible that programs written for cooperatively scheduled lisps that have run reliably for a long time have done so by accident: resource-contention issues tend to be timing-sensitive, and decoupling thread scheduling from lisp program execution affects timing. I know that there is or was code in both Clozure CL and commercial MCL that was written under the explicit assumption that certain sequences of open-coded operations were uninterruptable; it's certainly possible that the same assumptions have been made (explicitly or otherwise) by application developers.

7.5. Background Terminal Input

7.5.1. Overview

Unless and until Clozure CL provides alternatives (via window streams, telnet streams, or some other mechanism) all lisp processes share a common *TERMINAL-IO* stream (and therefore share *DEBUG-IO*, *QUERY-IO*, and other standard and internal interactive streams.)

It's anticipated that most lisp processes other than the "Initial" process run mostly in the background. If a background process writes to the output side of *TERMINAL-IO*, that may be a little messy and a little confusing to the user, but it shouldn't really be catastrophic. All I/O to Clozure CL's buffered streams goes thru a locking mechanism that prevents the worst kinds of resource-contention problems.

Although the problems associated with terminal output from multiple processes may be mostly cosmetic, the question of which process receives input from the terminal is likely to be a great deal more important. The stream locking mechanisms can make a confusing situation even worse: competing processes may "steal" terminal input from each other unless locks are held longer than they otherwise need to be, and locks can be held longer than they need to be (as when a process is merely waiting for input to become available on an underlying file descriptor).

Even if background processes rarely need to intentionally read input from the terminal, they may still need to do so in response to errors or other unanticipated situations. There are tradeoffs involved in any solution to this problem. The protocol described below allows background processes which follow it to reliably prompt for and receive terminal input. Background processes which attempt to receive terminal input without following this protocol will likely hang indefinitely while attempting to do so. That's certainly a harsh tradeoff, but since attempts to read terminal input without following this protocol only worked some of the time anyway, it doesn't seem to be an unreasonable one.

In the solution described here (and introduced in Clozure CL 0.9), the internal stream used to provide terminal input is always locked by some process (the "owning" process.) The initial process (the process that typically runs the read-eval-print loop) owns that stream when it's first created. By using the macro WITH-TERMINAL-INPUT, background processes can temporarily obtain ownership of the terminal and relinquish ownership to the previous owner when they're done with it.

In Clozure CL, BREAK, ERROR, CERROR, Y-OR-N-P, YES-OR-NO-P, and CCL:GET-STRING- FROM-USER are all defined in terms of WITH-TERMINAL-INPUT, as are the :TTY user-interfaces to STEP and INSPECT.

7.5.2. An example

? Welcome to Clozure CL Version (Beta: linux) 0.9!
?

? (process-run-function "sleeper" #'(lambda () (sleep 5) (break "broken")))
#<PROCESS sleeper(1) [Enabled] #x3063B33E>

?
;;
;; Process sleeper(1) needs access to terminal input.
;;
      

This example was run under ILISP; ILISP often gets confused if one tries to enter input and "point" doesn't follow a prompt. Entering a "simple" expression at this point gets it back in synch; that's otherwise not relevant to this example.

()
NIL
? (:y 1)
;;
;; process sleeper(1) now controls terminal input
;;
> Break in process sleeper(1): broken
> While executing: #<Anonymous Function #x3063B276>
> Type :GO to continue, :POP to abort.
> If continued: Return from BREAK.
Type :? for other options.
1 > :b
(30C38E30) : 0 "Anonymous Function #x3063B276" 52
(30C38E40) : 1 "Anonymous Function #x304984A6" 376
(30C38E90) : 2 "RUN-PROCESS-INITIAL-FORM" 340
(30C38EE0) : 3 "%RUN-STACK-GROUP-FUNCTION" 768
1 > :pop
;;
;; control of terminal input restored to process Initial(0)
;;
?
      

7.5.3. A more elaborate example.

If a background process ("A") needs access to the terminal input stream and that stream is owned by another background process ("B"), process "A" announces that fact, then waits until the initial process regains control.

? Welcome to Clozure CL Version (Beta: linux) 0.9!
?

? (process-run-function "sleep-60" #'(lambda () (sleep 60) (break "Huh?")))
#<PROCESS sleep-60(1) [Enabled] #x3063BF26>

? (process-run-function "sleep-5" #'(lambda () (sleep 5) (break "quicker")))
#<PROCESS sleep-5(2) [Enabled] #x3063D0A6>

?       ;;
;; Process sleep-5(2) needs access to terminal input.
;;
()
NIL

? (:y 2)
;;
;; process sleep-5(2) now controls terminal input
;;
> Break in process sleep-5(2): quicker
> While executing: #x3063CFDE>
> Type :GO to continue, :POP to abort.
> If continued: Return from BREAK.
Type :? for other options.
1 >     ;; Process sleep-60(1) will need terminal access when
;; the initial process regains control of it.
;;
()
NIL
1 > :pop
;;
;; Process sleep-60(1) needs access to terminal input.
;;
;;
;; control of terminal input restored to process Initial(0)
;;

? (:y 1)
;;
;; process sleep-60(1) now controls terminal input
;;
> Break in process sleep-60(1): Huh?
> While executing: #x3063BE5E>
> Type :GO to continue, :POP to abort.
> If continued: Return from BREAK.
Type :? for other options.
1 > :pop
;;
;; control of terminal input restored to process Initial(0)
;;

?
      

7.5.4. Summary

This scheme is certainly not bulletproof: imaginative use of PROCESS-INTERRUPT and similar functions might be able to defeat it and deadlock the lisp, and any scenario where several background processes are clamoring for access to the shared terminal input stream at the same time is likely to be confusing and chaotic. (An alternate scheme, where the input focus was magically granted to whatever thread the user was thinking about, was considered and rejected due to technical limitations.)

The longer-term fix would probably involve using network or window-system streams to give each process unique instances of *TERMINAL-IO*.

Existing code that attempts to read from *TERMINAL-IO* from a background process will need to be changed to use WITH-TERMINAL-INPUT. Since that code was probably not working reliably in previous versions of Clozure CL, this requirement doesn't seem to be too onerous.

Note that WITH-TERMINAL-INPUT both requests ownership of the terminal input stream and promises to restore that ownership to the initial process when it's done with it. An ad hoc use of READ or READ-CHAR doesn't make this promise; this is the rationale for the restriction on the :Y command.

7.6. The Threads which Clozure CL Uses for Its Own Purposes

In the "tty world", Clozure CL starts out with 2 lisp-level threads:

? :proc
1 : -> listener     [Active]
0 :    Initial      [Active]
    

If you look at a running Clozure CL with a debugging tool, such as GDB, or Apple's Thread Viewer.app, you'll see an additional kernel-level thread on Darwin; this is used by the Mach exception-handling mechanism.

The initial thread, conveniently named "initial", is the one that was created by the operating system when it launched Clozure CL. It maps the heap image into memory, does some Lisp-level initialization, and, when the Cocoa IDE isn't being used, creates the thread "listener", which runs the top-level loop that reads input, evaluates it, and prints the result.

After the listener thread is created, the initial thread does "housekeeping": it sits in a loop, sleeping most of the time and waking up occasionally to do "periodic tasks". These tasks include forcing output on specified interactive streams, checking for and handling control-C interrupts, etc. Currently, those tasks also include polling for the exit status of external processes and handling some kinds of I/O to and from those processes.

In this environment, the initial thread does these "housekeeping" activities as necessary, until ccl:quit is called; quitting interrupts the initial thread, which then ends all other threads in as orderly a fashion as possible and calls the C function #_exit.

The short-term plan is to handle each external-process in a dedicated thread; the worst-case behavior of the current scheme can involve busy-waiting and excessive CPU utilization while waiting for an external process to terminate in some cases.

The Cocoa features use more threads. Adding a Cocoa listener creates two threads:

      ? :proc
      3 : -> Listener     [Active]
      2 :    housekeeping  [Active]
      1 :    listener     [Active]
      0 :    Initial      [Active]
    

The Cocoa event loop has to run in the initial thread; when the event loop starts up, it creates a new thread to do the "housekeeping" tasks which the initial thread would do in the terminal-only mode. The initial thread then becomes the one to receive all Cocoa events from the window server; it's the only thread which can.

It also creates one "Listener" (capital-L) thread for each listener window, with a lifetime that lasts as long as the thread does. So, if you open a second listener, you'll see five threads all together:

      ? :proc
      4 : -> Listener-2   [Active]
      3 :    Listener     [Active]
      2 :    housekeeping  [Active]
      1 :    listener     [Active]
      0 :    Initial      [Active]
    

Unix signals, such as SIGINT (control-C), invoke a handler installed by the Lisp kernel. Although the OS doesn't make any specific guarantee about which thread will receive the signal, in practice, it seems to be the initial thread. The handler just sets a flag and returns; the housekeeping thread (which may be the initial thread, if Cocoa's not being used) will check for the flag and take whatever action is appropriate to the signal.

In the case of SIGINT, the action is to enter a break loop, by calling on the thread being interrupted. When there's more than one Lisp listener active, it's not always clear what thread that should be, since it really depends on the user's intentions, which there's no way to divine programmatically. To make its best guess, the handler first checks whether the value of ccl:*interactive-abort-process* is a thread, and, if so, uses it. If that fails, it chooses the thread which currently "owns" the default terminal input stream; see .

In the bleeding-edge version of the Cocoa support which is based on Hemlock, an Emacs-like editor, each editor window has a dedicated thread associated with it. When a keypress event comes in which affects that specific window the initial thread sends it to the window's dedicated thread. The dedicated thread is responsible for trying to interpret keypresses as Hemlock commands, applying those commands to the active buffer; it repeats this in a loop, until the window closes. The initial thread handles all other events, such as mouse clicks and drags.

This thread-per-window scheme makes many things simpler, including the process of entering a "recursive command loop" in commands like "Incremental Search Forward", etc. (It might be possible to handle all Hemlock commands in the Cocoa event thread, but these "recursive command loops" would have to maintain a lot of context/state information; threads are a straightforward way of maintaining that information.)

Currently (August 2004), when a dedicated thread needs to alter the contents of the buffer or the selection, it does so by invoking methods in the initial thread, for synchronization purposes, but this is probably overkill and will likely be replaced by a more efficient scheme in the future.

The per-window thread could probably take more responsibility for drawing and handling the screen than it currently does; -something- needs to be done to buffer screen updates a bit better in some cases: you don't need to see everything that happens during something like indentation; you do need to see the results...

When Hemlock is being used, listener windows are editor windows, so in addition to each "Listener" thread, you should also see a thread which handles Hemlock command processing.

The Cocoa runtime may make additional threads in certain special situations; these threads usually don't run lisp code, and rarely if ever run much of it.

7.7. Threads Dictionary

[Function]

all-processes => result
Obtain a fresh list of all known Lisp threads.

Values:

result---a list of all lisp processes (threads) known to Clozure CL.

Description:

Returns a list of all lisp processes (threads) known to Clozure CL as of the precise instant it's called. It's safe to traverse this list and to modify the cons cells that comprise that list (it's freshly consed.) Since other threads can create and kill threads at any time, there's generally no way to get an "accurate" list of all threads, and (generally) no sense in which such a list can be accurate.

[Function]

make-process name &key persistent priority class initargs stack-size vstack-size tstack-size initial-bindings use-standard-initial-bindings => process
Creates and returns a new process.

Arguments and Values:

name---a string, used to identify the process.

persistent---if true, requests that information about the process be retained by SAVE-APPLICATION so that an equivalent process can be restarted when a saved image is run. The default is nil.

priority---ignored. It shouldn't be ignored of course, but there are complications on some platforms. The default is 0.

class---the class of process object to create; should be a subclass of CCL:PROCESS. The default is CCL:PROCESS.

initargs---Any additional initargs to pass to MAKE-INSTANCE. The default is ().

stack-size---the size, in bytes, of the newly-created process's control stack; used for foreign function calls and to save function return address context. The default is CCL:*DEFAULT-CONTROL-STACK-SIZE*.

vstack-size---the size, in bytes, of the newly-created process's value stack; used for lisp function arguments, local variables, and other stack-allocated lisp objects. The default is CCL:*DEFAULT-VALUE-STACK-SIZE*.

tstack-size---the size, in bytes, of the newly-created process's temp stack; used for the allocation of dynamic-extent objects. The default is CCL:*DEFAULT-TEMP-STACK-SIZE*.

use-standard-initial-bindings---when true, the global "standard initial bindings" are put into effect in the new thread before. See DEF-STANDARD-INITIAL-BINDING. "standard" initial bindings are put into effect before any bindings specified by :initial-bindings are. The default is t. This option is deprecated: the correct behavior of many Clozure CL components depends on thread-local bindings of many special variables being in effect.

initial-bindings---an alist of (symbol . valueform) pairs, which can be used to initialize special variable bindings in the new thread. Each valueform is used to compute the value of a new binding of symbol in the execution environment of the newly-created thread. The default is nil.

process---the newly-created process.

Description:

Creates and returns a new lisp process (thread) with the specified attributes. process will not begin execution immediately; it will need to be preset (given an initial function to run, as by process-preset) and enabled (allowed to execute, as by process-enable) before it's able to actually do anything.

If valueform is a function, it is called, with no arguments, in the execution environment of the newly-created thread; the primary value it returns is used for the binding of the corresponding symbol.

Otherwise, valueform is evaluated in the execution environment of the newly-created thread, and the resulting value is used.

[Function]

process-suspend process => result
Suspends a specified process.

Arguments and Values:

process---a lisp process (thread).

result---T if process had been runnable and is now suspended; NIL otherwise. That is, T if process's process-suspend-count transitioned from 0 to 1.

Description:

Suspends process, preventing it from running, and stopping it if it was already running. This is a fairly expensive operation, because it involves a few calls to the OS. It also risks creating deadlock if used improperly, for instance, if the process being suspended owns a lock or other resource which another process will wait for.

Each call to process-suspend must be reversed by a matching call to process-resume before process is able to run. What process-suspend actually does is increment the process-suspend-count of process.

A process can't suspend itself, though this once worked and this documentation claimed has claimed that it did.

Notes:

process-suspend was previously called process-disable. process-enable now names a function for which there is no obvious inverse, so process-disable is no longer defined.

[Function]

process-resume process => result
Resumes a specified process which had previously been suspended by process-suspend.

Arguments and Values:

process---a lisp process (thread).

result---T if process had been suspended and is now runnable; NIL otherwise. That is, T if process's process-suspend-count transitioned from to 0.

Description:

Undoes the effect of a previous call to process-suspend; if all such calls are undone, makes the process runnable. Has no effect if the process is not suspended. What process-resume actually does is decrement the process-suspend-count of process, to a minimum of 0.

Notes:

This was previously called PROCESS-ENABLE; process-enable now does something slightly different.

[Function]

process-suspend-count process => result
Returns the number of currently-pending suspensions applicable to a given process.

Arguments and Values:

process---a lisp process (thread).

result---The number of "outstanding" process-suspend calls on process, or NIL if process has expired.

Description:

An "outstanding" process-suspend call is one which has not yet been reversed by a call to process-resume. A process expires when its initial function returns, although it may later be reset.

A process is runnable when it has a process-suspend-count of 0, has been preset as by process-preset, and has been enabled as by process-enable. Newly-created processes have a process-suspend-count of 0.

[Function]

process-preset process function &rest args => result
Sets the initial function and arguments of a specified process.

Arguments and Values:

process---a lisp process (thread).

function---a function, designated by itself or by a symbol which names it.

args---a list of values, appropriate as arguments to function.

result---undefined.

Description:

Typically used to initialize a newly-created or newly-reset process, setting things up so that when process becomes enabled, it will begin execution by applying function to args. process-preset does not enable process, although a process must be process-preset before it can be enabled. Processes are normally enabled by process-enable.

[Function]

process-enable process &optional timeout
Begins executing the initial function of a specified process.

Arguments and Values:

process---a lisp process (thread).

timeout---a time interval in seconds. May be any non-negative real number the floor of which fits in 32 bits. The default is 1.

result---undefined.

Description:

Tries to begin the execution of process. An error is signaled if process has never been process-preset. Otherwise, process invokes its initial function.

process-enable attempts to synchronize with process, which is presumed to be reset or in the act of resetting itself. If this attempt is not successful within the time interval specified by timeout, a continuable error is signaled, which offers the opportunity to continue waiting.

A process cannot meaningfully attempt to enable itself.

Notes:

It would be nice to have more discussion of what it means to synchronize with the process.

[Function]

process-run-function process-specifier function &rest args => process
Creates a process, presets it, and enables it.

Arguments and Values:

name---a string, used to identify the process. Passed to make-process.

function---a function, designated by itself or by a symbol which names it. Passed to preset-process.

persistent---a boolean, passed to make-process.

priority---ignored.

class---a subclass of CCL:PROCESS. Passed to make-process.

initargs---a list of any additional initargs to pass to make-process.

stack-size---a size, in bytes. Passed to make-process.

vstack-size---a size, in bytes. Passed to make-process.

tstack-size---a size, in bytes. Passed to make-process.

process---the newly-created process.

Description:

Creates a lisp process (thread) via make-process, presets it via process-preset, and enables it via process-enable. This means that process will immediately begin to execute. process-run-function is the simplest way to create and run a process.

[Function]

process-interrupt process function &rest args => result
Arranges for the target process to invoke a specified function at some point in the near future, and then return to what it was doing.

Arguments and Values:

process---a lisp process (thread).

function---a function.

args---a list of values, appropriate as arguments to function.

result---the result of applying function to args if process is the current-process, otherwise NIL.

Description:

Arranges for process to apply function to args at some point in the near future (interrupting whatever process was doing.) If function returns normally, process resumes execution at the point at which it was interrupted.

process must be in an enabled state in order to respond to a process-interrupt request. It's perfectly legal for a process to call process-interrupt on itself.

process-interrupt uses asynchronous POSIX signals to interrupt threads. If the thread being interrupted is executing lisp code, it can respond to the interrupt almost immediately (as soon as it has finished pseudo-atomic operations like consing and stack-frame initialization.)

If the interrupted thread is blocking in a system call, that system call is aborted by the signal and the interrupt is handled on return.

It is still difficult to reliably interrupt arbitrary foreign code (that may be stateful or otherwise non-reentrant); the interrupt request is handled when such foreign code returns to or enters lisp.

Notes:

It would probably be better for result to always be NIL, since the present behavior is inconsistent.

Process-interrupt works by sending signals between threads, via the C function #_pthread_signal. It could be argued that it should be done in one of several possible other ways under Darwin, to make it practical to asynchronously interrupt things which make heavy use of the Mach nanokernel.

[Variable]

*CURRENT-PROCESS*
Bound in each process, to that process itself.

Value Type:

A lisp process (thread).

Initial Value:

Bound separately in each process, to that process itself.

Description:

Used when lisp code needs to find out what process it is executing in. Shouldn't be set by user code.

See Also:
all-processes

[Function]

process-reset process &optional kill-option => result
Causes a specified process to cleanly exit from any ongoing computation.

Arguments and Values:

process---a lisp process (thread).

kill-option---an internal argument, must be nil.

result---undefined.

Description:

Causes process to cleanly exit from any ongoing computation and enter a state where it can be process-preset. This is implemented by signaling a condition of type PROCESS-RESET; user-defined condition handlers should generally refrain from attempting to handle conditions of this type.

The kill-option argument is for internal use only and should not be specified by user code

A process can meaningfully reset itself.

There is in general no way to know precisely when process has completed the act of resetting or killing itself; a process which has either entered the limbo of the reset state or exited has few ways of communicating either fact. process-enable can reliably determine when a process has entered the "limbo of the reset state", but can't predict how long the clean exit from ongoing computation might take: that depends on the behavior of unwind-protect cleanup forms, and of the OS scheduler.

Resetting a process other than *current-process* involves the use of process-interrupt.

[Function]

process-reset-and-enable process => result
Reset and enable the specified process, which may not be the current process.

Arguments and Values:

process---a lisp process (thread), which may not be the current process.

result---undefined.

Description:

Equivalent to calling (process-reset process) and (process-enable process).

[Function]

process-kill process => result
Causes a specified process to cleanly exit from any ongoing computation, and then exit.

Arguments and Values:

process---a lisp process (thread).

result---undefined.

Description:

Causes process to cleanly exit from any ongoing computation, and then exit.

[Function]

process-abort process &optional condition => NIL
Causes a specified process to process an abort condition, as if it had invoked abort.

Arguments and Values:

process---a lisp process (thread).

condition---a lisp condition. The default is NIL.

Description:

Entirely equivalent to calling (process-interrupt process (lambda () (abort condition))). Causes process to transfer control to the applicable handler or restart for abort.

If condition is non-NIL, process-abort does not consider any handlers which are explicitly bound to conditions other than condition.

[Variable]

*TICKS-PER-SECOND*
Bound to the clock resolution of the OS scheduler.

Value Type:

A positive integer.

Initial Value:

The clock resolution of the OS scheduler. Currently, both LinuxPPC and DarwinPPC yield an initial value of 100.

Description:

This value is ordinarily of marginal interest at best, but, for backward compatibility, some functions accept timeout values expressed in "ticks". This value gives the number of ticks per second.

[Function]

process-whostate process => whostate
Returns a string which describes the status of a specified process.

Description:

This information is primarily for the benefit of debugging tools. whostate is a terse report on what process is doing, or not doing, and why.

If the process is currently waiting in a call to process-wait or process-wait-with-timeout, its process-whostate will be the value which was passed to that function as whostate.

Notes:

This should arguably be SETFable, but doesn't seem to ever have been.

[Function]

process-allow-schedule
Used for cooperative multitasking; probably never necessary.

Description:

Advises the OS scheduler that the current thread has nothing useful to do and that it should try to find some other thread to schedule in its place. There's almost always a better alternative, such as waiting for some specific event to occur. For example, you could use a lock or semaphore.

Notes:

This is a holdover from the days of cooperative multitasking. All modern general-purpose operating systems use preemptive multitasking.

[Function]

process-wait whostate function &rest args => result
Causes the current lisp process (thread) to wait for a given predicate to return true.

Arguments and Values:

whostate---a string, which will be the value of process-whostate while the process is waiting.

function---a function, designated by itself or by a symbol which names it.

args---a list of values, appropriate as arguments to function.

result---NIL.

Description:

Causes the current lisp process (thread) to repeatedly apply function to args until the call returns a true result, then returns NIL. After each failed call, yields the CPU as if by process-allow-schedule.

As with process-allow-schedule, it's almost always more efficient to wait for some specific event to occur; this isn't exactly busy-waiting, but the OS scheduler can do a better job of scheduling if it's given the relevant information. For example, you could use a lock or semaphore.

[Function]

process-wait-with-timeout whostate ticks function args => result
Causes the current thread to wait for a given predicate to return true, or for a timeout to expire.

Arguments and Values:

whostate---a string, which will be the value of process-whostate while the process is waiting.

ticks---either a positive integer expressing a duration in "ticks" (see *ticks-per-second*), or NIL.

function---a function, designated by itself or by a symbol which names it.

args---a list of values, appropriate as arguments to function.

result---T if process-wait-with-timeout returned because its function returned true, or NIL if it returned because the duration ticks has been exceeded.

Description:

If ticks is NIL, behaves exactly like process-wait, except for returning T. Otherwise, function will be tested repeatedly, in the same kind of test/yield loop as in process-wait until either function returns true, or the duration ticks has been exceeded.

Having already read the descriptions of process-allow-schedule and process-wait, the astute reader has no doubt anticipated the observation that better alternatives should be used whenever possible.

[Macro]

without-interrupts &body body => result
Evaluates its body in an environment in which process-interrupt requests are deferred.

Arguments and Values:

body---an implicit progn.

result---the primary value returned by body.

Description:

Executes body in an environment in which process-interrupt requests are deferred. As noted in the description of process-interrupt, this has nothing to do with the scheduling of other threads; it may be necessary to inhibit process-interrupt handling when (for instance) modifying some data structure (for which the current thread holds an appropriate lock) in some manner that's not reentrant.

[Macro]

with-interrupts-enabled &body body => result
Evaluates its body in an environment in which process-interrupt requests have immediate effect.

Arguments and Values:

body---an implicit progn.

result---the primary value returned by body.

Description:

Executes body in an environment in which process-interrupt requests have immediate effect.

[Function]

make-lock &optional name => lock
Creates and returns a lock object, which can be used for synchronization between threads.

Arguments and Values:

name---any lisp object; saved as part of lock. Typically a string or symbol which may appear in the process-whostates of threads which are waiting for lock.

lock---a newly-allocated object of type CCL:LOCK.

Description:

Creates and returns a lock object, which can be used to synchronize access to some shared resource. lock is initially in a "free" state; a lock can also be "owned" by a thread.

[Macro]

with-lock-grabbed (lock) &body body
Waits until a given lock can be obtained, then evaluates its body with the lock held.

Arguments and Values:

lock---an object of type CCL:LOCK.

body---an implicit progn.

result---the primary value returned by body.

Description:

Waits until lock is either free or owned by the calling thread, then executes body with the lock owned by the calling thread. If lock was free when with-lock-grabbed was called, it is restored to a free state after body is executed.

[Function]

grab-lock lock
Waits until a given lock can be obtained, then obtains it.

Arguments and Values:

lock---an object of type CCL:LOCK.

Description:

Blocks until lock is owned by the calling thread.

The macro with-lock-grabbed could be defined in terms of grab-lock and release-lock, but it is actually implemented at a slightly lower level.

[Function]

release-lock lock
Relinquishes ownership of a given lock.

Arguments and Values:

lock---an object of type CCL:LOCK.

Description:

Signals an error of type CCL:LOCK-NOT-OWNER if lock is not already owned by the calling thread; otherwise, undoes the effect of one previous grab-lock. If this means that release-lock has now been called on lock the same number of times as grab-lock has, lock becomes free.

[Function]

try-lock lock => result
Obtains the given lock, but only if it is not necessary to wait for it.

Arguments and Values:

lock---an object of type CCL:LOCK.

result---T if lock has been obtained, or NIL if it has not.

Description:

Tests whether lock can be obtained without blocking - that is, either lock is already free, or it is already owned by *current-process*. If it can, causes it to be owned by the calling lisp process (thread) and returns T. Otherwise, the lock is already owned by another thread and cannot be obtained without blocking; NIL is returned in this case.

[Function]

make-read-write-lock => read-write-lock
Creates and returns a read-write lock, which can be used for synchronization between threads.

Arguments and Values:

read-write-lock---a newly-allocated object of type CCL:READ-WRITE-LOCK.

Description:

Creates and returns an object of type CCL::READ-WRITE-LOCK. A read-write lock may, at any given time, belong to any number of lisp processes (threads) which act as "readers"; or, it may belong to at most one process which acts as a "writer". A read-write lock may never be held by a reader at the same time as a writer. Initially, read-write-lock has no readers and no writers.

Notes:

There probably should be some way to atomically "promote" a reader, making it a writer without releasing the lock, which could otherwise cause delay.

[Macro]

with-read-lock (read-write-lock) &body body => result
Waits until a given lock is available for read-only access, then evaluates its body with the lock held.

Arguments and Values:

read-write-lock---an object of type CCL:READ-WRITE-LOCK.

body---an implicit progn.

result---the primary value returned by body.

Description:

Waits until read-write-lock has no writer, ensures that *current-process* is a reader of it, then executes body.

After executing body, if *current-process* was not a reader of read-write-lock before with-read-lock was called, the lock is released. If it was already a reader, it remains one.

[Macro]

with-write-lock (read-write-lock) &body body
Waits until the given lock is available for write access, then executes its body with the lock held.

Arguments and Values:

read-write-lock---an object of type CCL:READ-WRITE-LOCK.

body---an implicit progn.

result---the primary value returned by body.

Description:

Waits until read-write-lock has no readers and no writer other than *current-process*, then ensures that *current-process* is the writer of it. With the lock held, executes body.

After executing body, if *current-process* was not the writer of read-write-lock before with-write-lock was called, the lock is released. If it was already the writer, it remains the writer.

[Function]

make-semaphore => semaphore
Creates and returns a semaphore, which can be used for synchronization between threads.

Arguments and Values:

semaphore---a newly-allocated object of type CCL:SEMAPHORE.

Description:

Creates and returns an object of type CCL:SEMAPHORE. A semaphore has an associated "count" which may be incremented and decremented atomically; incrementing it represents sending a signal, and decrementing it represents handling that signal. semaphore has an initial count of 0.

[Function]

signal-semaphore semaphore => result
Atomically increments the count of a given semaphore.

Arguments and Values:

semaphore---an object of type CCL:SEMAPHORE.

result---an integer representing an error identifier which was returned by the underlying OS call.

Description:

Atomically increments semaphore's "count" by 1; this may enable a waiting thread to resume execution.

Notes:

result should probably be interpreted and acted on by signal-semaphore, because it is not likely to be meaningful to a lisp program, and the most common cause of failure is a type error.

[Function]

wait-on-semaphore semaphore => result
Waits until the given semaphore has a positive count which can be atomically decremented.

Arguments and Values:

semaphore---an object of type CCL:SEMAPHORE.

result---an integer representing an error identifier which was returned by the underlying OS call.

Description:

Waits until semaphore has a positive count that can be atomically decremented; this will succeed exactly once for each corresponding call to SIGNAL-SEMAPHORE.

Notes:

result should probably be interpreted and acted on by wait-on-semaphore, because it is not likely to be meaningful to a lisp program, and the most common cause of failure is a type error.

[Function]

timed-wait-on-semaphore semaphore timeout => result
Waits until the given semaphore has a positive count which can be atomically decremented, or until a timeout expires.

Arguments and Values:

semaphore---An object of type CCL:SEMAPHORE.

timeout---a time interval in seconds. May be any non-negative real number the floor of which fits in 32 bits. The default is 1.

result---T if timed-wait-on-semaphore returned because it was able to decrement the count of semaphore; NIL if it returned because the duration timeout has been exceeded.

Description:

Waits until semaphore has a positive count that can be atomically decremented, or until the duration timeout has elapsed.

[Function]

process-input-wait fd &optional timeout
Waits until input is available on a given file-descriptor.

Arguments and Values:

fd---a file descriptor, which is a non-negative integer used by the OS to refer to an open file, socket, or similar I/O connection. See ccl::stream-device.

timeout---either NIL or a time interval in milliseconds. Must be a non-negative integer. The default is NIL.

Description:

Wait until input is available on fd. This uses the select() system call, and is generally a fairly efficient way of blocking while waiting for input. More accurately, process-input-wait waits until it's possible to read from fd without blocking, or until timeout, if it is not NIL, has been exceeded.

Note that it's possible to read without blocking if the file is at its end - although, of course, the read will return zero bytes.

Notes:

process-input-wait has a timeout parameter, and process-output-wait does not. This inconsistency should probably be corrected.

[Function]

process-output-wait fd &optional timeout
Waits until output is possible on a given file descriptor.

Arguments and Values:

fd---a file descriptor, which is a non-negative integer used by the OS to refer to an open file, socket, or similar I/O connection. See ccl::stream-device.

timeout---either NIL or a time interval in milliseconds. Must be a non-negative integer. The default is NIL.

Description:

Wait until output is possible on fd or until timeout, if it is not NIL, has been exceeded. This uses the select() system call, and is generally a fairly efficient way of blocking while waiting to output.

If process-output-wait is called on a network socket which has not yet established a connection, it will wait until the connection is established. This is an important use, often overlooked.

Notes:

process-input-wait has a timeout parameter, and process-output-wait does not. This inconsistency should probably be corrected.

[Macro]

with-terminal-input &body body => result
Executes its body in an environment with exclusive read access to the terminal.

Arguments and Values:

body---an implicit progn.

result---the primary value returned by body.

Description:

Requests exclusive read access to the standard terminal stream, *terminal-io*. Executes body in an environment with that access.

[Variable]

*REQUEST-TERMINAL-INPUT-VIA-BREAK*
Controls how attempts to obtain ownership of terminal input are made.

Value Type:

A boolean.

Initial Value:

NIL.

Description:

Controls how attempts to obtain ownership of terminal input are made. When NIL, a message is printed on *TERMINAL-IO*; it's expected that the user will later yield control of the terminal via the :Y toplevel command. When T, a BREAK condition is signaled in the owning process; continuing from the break loop will yield the terminal to the requesting process (unless the :Y command was already used to do so in the break loop.)

[Toplevel Command]

(:y p)
Yields control of terminal input to a specified lisp process (thread).

Arguments and Values:

p---a lisp process (thread), designated either by an integer which matches its process-serial-number, or by a string which is equal to its process-name.

Description:

:Y is a toplevel command, not a function. As such, it can only be used interactively, and only from the initial process.

The command yields control of terminal input to the process p, which must have used with-terminal-input to request access to the terminal input stream.

[Function]

join-process process &optional default => values
Waits for a specified process to complete and returns the values that that process's initial function returned.

Arguments and Values:

process---a process, typically created by process-run-function or by make-process

default---A default value to be returned if the specified process doesn't exit normally.

values---The values returned by the specified process's initial function if that function returns, or the value of the default argument, otherwise.

Description:

Waits for the specified process to terminate. If the process terminates "normally" (if its initial function returns), returns the values that that initial function returnes. If the process does not terminate normally (e.g., if it's terminated via process-kill and a default argument is provided, returns the value of that default argument. If the process doesn't terminate normally and no default argument is provided, signals an error.

A process can't successfully join itself, and only one process can successfully receive notification of another process's termination.

Chapter 8. Programming with Sockets

8.1. Overview

Clozure CL supports the socket abstraction for interprocess communication. A socket represents a connection to another process, typically (but not necessarily) a TCP/IP network connection to a client or server running on some other machine on the network.

All symbols mentioned in this chapter are exported from the CCL package. As of version 0.13, these symbols are additionally exported from the OPENMCL-SOCKET package.

Clozure CL supports three types of sockets: TCP sockets, UDP sockets, and Unix-domain sockets. This should be enough for all but the most esoteric network situations. All sockets are created by make-socket. The type of socket depends on the arguments to it, as follows:

tcp-stream

A buffered bi-directional stream over a TCP/IP connection. tcp-stream is a subclass of stream, and you can read and write to it using all the usual stream functions. Created by (make-socket :address-family :internet :type :stream :connect :active ...) or by (accept-connection ...).

file-socket-stream

A buffered bi-directional stream over a "UNIX domain" connection. file-socket-stream is a subclass of stream, and you can read and write to it using all the usual stream functions. Created by (make-socket :address-family :file :type :stream :connect :active ...) or by (accept-connection ...),

listener-socket

A passive socket used to listen for incoming TCP/IP connections on a particular port. A listener-socket is not a stream. It doesn't support I/O. It can only be used to create new tcp-streams by accept-connection. Created by (make-socket :type :stream :connect :passive ...)

file-listener-socket

A passive socket used to listen for incoming UNIX domain connections named by a file in the local filesystem. A listener-socket is not a stream. It doesn't support I/O. It can only be used to create new file-socket-streams by accept-connection. Created by (make-socket :address-family :file :type :stream :connect :passive ...)

udp-socket

A socket representing a packet-based UDP/IP connection. A udp-socket supports I/O but it is not a stream. Instead, you must use the special functions send-to and receive-from to read and write to it. Created by (make-socket :type :datagram ...)

8.2. Sockets Dictionary

[Function]

make-socket &key address-family type connect eol format remote-host remote-port local-host local-port local-filename remote-filename keepalive reuse-address nodelay broadcast linger backlog input-timeout output-timeout connect-timeout auto-close deadline

Arguments and Values:

address-family---The address/protocol family of this socket. Currently only :internet (the default), meaning IP, and :file, referring to UNIX domain addresses, are supported.

type---One of :stream (the default) to request a connection-oriented socket, or :datagram to request a connectionless socket. The default is :stream.

connect---This argument is only relevant to sockets of type :stream. One of :active (the default) to request a :passive to request a file or TCP listener socket.

eol---This argument is currently ignored (it is accepted for compatibility with Franz Allegro).

format---One of :text (the default), :binary, or :bivalent. This argument is ignored for :stream sockets for now, as :stream sockets are currently always bivalent (i.e. they support both character and byte I/O). For :datagram sockets, this argument is ignored (the format of a datagram socket is always :binary).

remote-host---Required for TCP streams, it specifies the host to connect to (in any format acceptable to lookup-hostname). Ignored for listener sockets. For UDP sockets, it can be used to specify a default host for subsequent calls to send-to or receive-from.

remote-port---Required for TCP streams, it specifies the port to connect to (in any format acceptable to lookup-port). Ignored for listener sockets. For UDP sockets, it can be used to specify a default port for subsequent calls to for subsequent calls to send-to or receive-from.

remote-filename---Required for file-socket streams, it specifies the name of a file in the local filesystem (e.g., NOT mounted via NFS, AFP, SMB, ...) which names and controls access to a UNIX-domain socket.

local-host---Allows you to specify a local host address for a listener or UDP socket, for the rare case where you want to restrict connections to those coming to a specific local address for security reasons.

local-port---Specify a local port for a socket. Most useful for listener sockets, where it is the port on which the socket will listen for connections.

local-filename---Required for file-listener-sockets. Specifies the name of a file in the local filesystem which is used to name a UNIX-domain socket. The actual filesystem file should not previously exist when the file-listener-socket is created; its parent directory should exist and be writable by the caller. The file used to name the socket will be deleted when the file-listener-socket is closed.

keepalive---If true, enables the periodic transmission of "keepalive" messages.

reuse-address---If true, allows the reuse of local ports in listener sockets, overriding some TCP/IP protocol specifications. You will need this if you are debugging a server..

nodelay---If true, disables Nagle's algorithm, which tries to minimize TCP packet fragmentation by introducing transmission delays in the absence of replies. Try setting this if you are using a protocol which involves sending a steady stream of data with no replies and are seeing significant degradations in throughput.

broadcast---If true, requests permission to broadcast datagrams on a UDP socket.

linger---If specified and non-nil, should be the number of seconds the OS is allowed to wait for data to be pushed through when a close is done. Only relevant for TCP sockets.

backlog---For a listener socket, specifies the number of connections which can be pending but not accepted. The default is 5, which is also the maximum on some operating systems.

input-timeout---The number of seconds before an input operation times out. Must be a real number between zero and one million. If an input operation takes longer than the specified number of seconds, an input-timeout error is signalled. (see Section 10.1.4, “Stream Timeouts and Deadlines”)

output-timeout---The number of seconds before an output operation times out. Must be a real number between zero and one million. If an output operation takes longer than the specified number of seconds, an output-timeout error is signalled. (see Section 10.1.4, “Stream Timeouts and Deadlines”)

connect-timeout---The number of seconds before a connection attempt times out. [TODO: what are acceptable values?] If a connection attempt takes longer than the specified number of seconds, a socket-error is signalled. This can be useful if the specified interval is shorter than the interval that the OS's socket layer imposes, which is sometimes a minute or two.

auto-close---When non-nil, any resulting socket stream will be closed when the GC can prove that the stream is unreferenced. This is done via CCL's termination mechanism [TODO add xref].

deadline---Specifies an absolute time in internal-time-units. If an I/O operation on the stream does not complete before the deadline then a COMMUNICATION-DEADLINE-EXPIRED error is signalled. A deadline takes precedence over any input/output timeouts that may be set. (see Section 10.1.4, “Stream Timeouts and Deadlines”)

Description:

Creates and returns a new socket

[Function]

accept-connection (socket listener-socket) &key wait

Arguments and Values:

socket---The listener-socket to listen on.

wait---If true (the default), and there are no connections waiting to be accepted, waits until one arrives. If false, returns NIL immediately.

Description:

Extracts the first connection on the queue of pending connections, accepts it (i.e. completes the connection startup protocol) and returns a new tcp-stream or file-socket-stream representing the newly established connection. The tcp stream inherits any properties of the listener socket that are relevant (e.g. :keepalive, :nodelay, etc.) The original listener socket continues to be open listening for more connections, so you can call accept-connection on it again.

[Function]

dotted-to-ipaddr dotted &key errorp

Arguments and Values:

dotted---A string representing an IP address in the "nn.nn.nn.nn" format

errorp---If true (the default) an error is signaled if dotted is invalid. If false, NIL is returned.

Description:

Converts a dotted-string representation of a host address to a 32-bit unsigned IP address.

[Function]

ipaddr-to-dotted ipaddr &key values

Arguments and Values:

ipaddr---A 32-bit integer representing an internet host address

values---If false (the default), returns a string in the form "nn.nn.nn.nn". If true, returns four values representing the four octets of the address as unsigned 8-bit integers.

Description:

Converts a 32-bit unsigned IP address into octets.

[Function]

ipaddr-to-hostname ipaddr &key ignore-cache

Arguments and Values:

ipaddr---a 32-bit integer representing an internet host address

ignore-cache---This argument is ignored (it is accepted for compatibility with Franz Allegro)

Description:

Converts a 32-bit unsigned IP address into a host name string

[Function]

lookup-hostname host

Arguments and Values:

host---Specifies the host. It can be either a host name string such as "clozure.com", or a dotted address string such as "192.168.0.1", or a 32-bit unsigned IP address such as 3232235521.

Description:

Converts a host spec in any of the acceptable formats into a 32-bit unsigned IP address

[Function]

lookup-port port protocol

Arguments and Values:

port---Specifies the port. It can be either a string, such as "http" or a symbol, such as :http, or an unsigned port number. Note that a string is case-sensitive. A symbol is lowercased before lookup.

protocol---Must be one of "tcp" or "udp".

Description:

Finds the port number for the specified port and protocol

[Function]

receive-from (socket udp-socket) size &key buffer extract offset

Arguments and Values:

socket---The socket to read from

size---Maximum number of bytes to read. If the packet is larger than this, any extra bytes are discarded.

buffer---If specified, must be an octet vector which will be used to read in the data. If not specified, a new buffer will be created (of type determined by socket-format).

extract---If true, the subsequence of the buffer corresponding only to the data read in is extracted and returned as the first value. If false (the default) the original buffer is returned even if it is only partially filled.

offset---Specifies the start offset into the buffer at which data is to be stored. The default is 0.

Description:

Reads a UDP packet from a socket. If no packets are available, waits for a packet to arrive. Returns four values:

  1. The buffer with the data

  2. The number of bytes read

  3. The 32-bit unsigned IP address of the sender of the data

  4. The port number of the sender of the data

[Function]

send-to (socket udp-socket) buffer size &key remote-host remote-port offset

Arguments and Values:

socket---The socket to write to

buffer---A vector containing the data to send. It must be an octet vector.

size---Number of octets to send

remote-host---The host to send the packet to, in any format acceptable to lookup-hostname. The default is the remote host specified in the call to make-socket.

remote-port---The port to send the packet to, in any format acceptable to lookup-port. The default is the remote port specified in the call to make-socket.

offset---The offset in the buffer where the packet data starts

Description:

Send a UDP packet over a socket.

[Function]

shutdown socket &key direction

Arguments and Values:

socket---The socket to shut down (typically a tcp-stream)

direction---One of :input to disallow further input, or :output to disallow further output.

Description:

Shuts down part of a bidirectional connection. This is useful if e.g. you need to read responses after sending an end-of-file signal.

[Function]

socket-os-fd socket

Arguments and Values:

socket---The socket

Description:

Returns the native OS's representation of the socket, or NIL if the socket is closed. On Unix, this is the Unix 'file descriptor', a small non-negative integer. Note that it is rather dangerous to mess around with tcp-stream fd's, as there is all sorts of buffering and asynchronous I/O going on above the OS level. listener-socket and udp-socket fd's are safer to mess with directly as there is less magic going on.

[Function]