Previous Section Next Section Table of Contents Glossary Index

Chapter 10. Streams

10.2. Creating Your Own Stream Classes with Gray Streams

10.2.1. Overview

This sect1 is still being written and revised, because it is woefully incomplete. The dictionary section currently only lists a couple functions. Caveat lector.

Gray streams are an extension to Common Lisp. They were proposed for standardization by David Gray (the astute reader now understands their name) quite some years ago, but not accepted, because they had not been tried sufficiently to find conceptual problems with them.

They have since been implemented by quite a few modern Lisp implementations. However, they do indeed have some inadequacies, and each implementation has addressed these in different ways. The situation today is that it's difficult to even find out how to get started using Gray streams. This is why standards are important.

Here's a list of some classes which you might wish for your new stream class to inherit from:

fundamental-stream
fundamental-input-stream
fundamental-output-stream
fundamental-character-stream
fundamental-binary-stream
fundamental-character-input-stream
fundamental-character-output-stream
fundamental-binary-input-stream
fundamental-binary-output-stream
ccl::buffered-stream-mixin
ccl::buffered-input-stream-mixin
ccl::buffered-output-stream-mixin
ccl::buffered-io-stream-mixin
ccl::buffered-character-input-stream-mixin
ccl::buffered-character-output-stream-mixin
ccl::buffered-character-io-stream-mixin
ccl::buffered-binary-input-stream-mixin
ccl::buffered-binary-output-stream-mixin
ccl::buffered-binary-io-stream-mixin
file-stream
file-input-stream
file-output-stream
file-io-stream
file-character-input-stream
file-character-output-stream
file-character-io-stream
file-binary-input-stream
file-binary-output-stream
file-binary-io-stream
ccl::fd-stream
ccl::fd-input-stream
ccl::fd-output-stream
ccl::fd-io-stream
ccl::fd-character-input-stream
ccl::fd-character-output-stream
ccl::fd-character-io-stream
ccl::fd-binary-input-stream
ccl::fd-binary-output-stream
ccl::fd-binary-io-stream

All of these are defined in ccl/level-1/l1-streams.lisp, except for the ccl:file-* ones, which are in ccl/level-1/l1-sysio.lisp.

According to the original Gray streams proposal, you should inherit from the most specific of the fundamental-* classes which applies. Using Clozure CL, though, if you want buffering for better performance, which, unless you know of some reason you wouldn't, you do, you should instead inherit from the appropriate ccl::buffered-* class The buffering you get this way is exactly the same as the buffering which is used on ordinary, non-Gray streams, and force-output will work properly on it.

Notice that -mixin suffix in the names of all the ccl::buffered-* classes? The suffix means that this class is not "complete" by itself; you still need to inherit from a fundamental-* stream, even if you also inherit from a *-mixin stream. You might consider making your own class like this. .... Except that they do inherit from the fundamental-* streams, that's weird.

If you want to be able to create an instance of your class with the :class argument to (open) and (with-open-file), you should make it inherit from one of the file-* classes. If you do this, it's not necessary to inherit from any of the other classes (though it won't hurt anything), since the file-* classes already do.

When you inherit from the file-* classes, you can use (call-next-method) in any of your methods to get the standard behavior. This is especially useful if you want to create a class which performs some simple filtering operation, such as changing everything to uppercase or to a different character encoding. If you do this, you will definitely need to specialize ccl::select-stream-class. Your method on ccl::stream-select-class should accept an instance of the class, but pay no attention to its contents, and return a symbol naming the class to actually be instantiated.

If you need to make your functionality generic across all the different types of stream, probably the best way to implement it is to make it a mixin, define classes with all the variants of input, output, io, character, and binary, which inherit both from your mixin and from the appropriate other class, then define a method on ccl::select-stream-class which chooses from among those classes.

Note that some of these classes are internal to the CCL package. If you try to inherit from those ones without the ccl:: prefix, you'll get an error which may confuse you, calling them "forward-referenced classes". That just means you used the wrong symbol, so add the prefix.

Here's a list of some generic functions which you might wish to specialize for your new stream class, and which ought to be documented at some point.

stream-direction stream =>
stream-device stream direction =>
stream-length stream &optional new =>
stream-position stream &optional new =>
streamp stream => boolean
stream-write-char output-stream char =>
stream-write-entire-string output-stream string =>
stream-read-char input-stream =>
stream-unread-char input-stream char =>
stream-force-output output-stream => nil
stream-maybe-force-output output-stream => nil
stream-finish-output output-stream => nil
stream-clear-output output-stream => nil
close stream &key abort => boolean
stream-fresh-line stream => t
stream-line-length stream => length
interactive-stream-p stream => boolean
stream-clear-input input-stream => nil
stream-listen input-stream => boolean
stream-filename stream => string
ccl::select-stream-class instance in-p out-p char-p => class

The following functions are standard parts of Common Lisp, but behave in special ways with regard to Gray streams.

open-stream-p stream => generalized-boolean
input-stream-p stream => generalized-boolean
output-stream-p stream => generalized-boolean
stream-element-type stream =>
stream-error-stream =>
open
close
with-open-file

Specifically, (open) and (with-open-file) accept a new keyword argument, :class, which may be a symbol naming a class; the class itself; or an instance of it. The class so given must be a subtype of 'stream, and an instance of it with no particular contents will be passed to ccl::select-stream-class to determine what class to actually instantiate.

The following are standard, and do not behave specially with regard to Gray streams, but probably should.

stream-external-format

10.2.2. Extending READ-SEQUENCE and WRITE-SEQUENCE

10.2.2.1. Overview

The "Gray Streams" API is based on an informal proposal that was made before ANSI CL adopted the READ-SEQUENCE and WRITE-SEQUENCE functions; as such, there is no "standard" way for the author of a Gray stream class to improve the performance of these functions by exploiting knowledge of the stream's internals (e.g., the buffering mechanism it uses.)

In the absence of any such knowledge, READ-SEQUENCE and WRITE-SEQUENCE are effectively just convenient shorthand for a loop which calls READ-CHAR/READ-BYTE/WRITE-CHAR/WRITE-BYTE as appropriate. The mechanism described below allows subclasses of FUNDAMENTAL-STREAM to define more specialized (and presumably more efficient) behavior.

10.2.2.2. Notes

READ-SEQUENCE and WRITE-SEQUENCE do a certain amount of sanity-checking and normalization of their arguments before dispatching to one of the methods above. If an individual method can't do anything particularly clever, CALL-NEXT-METHOD can be used to handle the general case.

10.2.2.3. Example

(defclass my-string-input-stream (fundamental-character-input-stream)
  ((string :initarg :string :accessor my-string-input-stream-string)
   (index :initform 0 :accessor my-string-input-stream-index)
   (length)))

(defmethod stream-read-vector ((stream my-string-input-stream) vector start end)
  (if (not (typep vector 'simple-base-string))
      (call-next-method)
      (with-slots (string index length)
	      (do* ((outpos start (1+ outpos)))
               ((or (= outpos end)
                    (= index length))
                outpos))
        (setf (schar vector outpos)
              (schar string index))
        (incf index)))))
	    

10.2.3. Multibyte I/O

All heap-allocated objects in Clozure CL that cannot contain pointers to lisp objects are represented as ivectors. Clozure CL provides low-level functions, and , to efficiently transfer data between buffered streams and ivectors. There's some overlap in functionality between the functions described here and the ANSI CL READ-SEQUENCE and WRITE-SEQUENCE functions.

As used here, the term "octet" means roughly the same thing as the term "8-bit byte". The functions described below transfer a specified sequence of octets between a buffered stream and an ivector, and don't really concern themselves with higher-level issues (like whether that octet sequence is within bounds or how it relates to the logical contents of the ivector.) For these reasons, these functions are generally less safe and more flexible than their ANSI counterparts.

10.2.4. Gray Streams Dictionary

[Generic Function]

stream-read-list stream list count

Arguments and Values:

stream---a stream, presumably a fundamental-input-stream.

list---a list. When a STREAM-READ-LIST method is called by READ-SEQUENCE, this argument is guaranteed to be a proper list.

count---a non-negative integer. When a STREAM-READ-LIST method is called by READ-SEQUENCE, this argument is guaranteed not to be greater than the length of the list.

Description:

Should try to read up to count elements from stream into the list list, returning the number of elements actually read (which may be less than count in case of a premature end-of-file.)

[Generic Function]

stream-write-list stream list count

Arguments and Values:

stream---a stream, presumably a fundamental-output-stream.

list---a list. When a STREAM-WRITE-LIST method is called by WRITE-SEQUENCE, this argument is guaranteed to be a proper list.

count---a non-negative integer. When a STREAM-WRITE-LIST method is called by WRITE-SEQUENCE, this argument is guaranteed not to be greater than the length of the list.

Description:

should try to write the first count elements of list to stream. The return value of this method is ignored.

[Generic Function]

stream-read-vector stream vector start end

Arguments and Values:

stream---a stream, presumably a fundamental-input-stream

vector---a vector. When a STREAM-READ-VECTOR method is called by READ-SEQUENCE, this argument is guaranteed to be a simple one-dimensional array.

start---a non-negative integer. When a STREAM-READ-VECTOR method is called by READ-SEQUENCE, this argument is guaranteed to be no greater than end and not greater than the length of vector.

end---a non-negative integer. When a STREAM-READ-VECTOR method is called by READ-SEQUENCE, this argument is guaranteed to be no less than end and not greater than the length of vector.

Description:

should try to read successive elements from stream into vector, starting at element start (inclusive) and continuing through element end (exclusive.) Should return the index of the vector element beyond the last one stored into, which may be less than end in case of premature end-of-file.

[Generic Function]

stream-write-vector stream vector start end

Arguments and Values:

stream---a stream, presumably a fundamental-output-stream

vector---a vector. When a STREAM-WRITE-VECTOR method is called by WRITE-SEQUENCE, this argument is guaranteed to be a simple one-dimensional array.

start---a non-negative integer. When a STREAM-WRITE-VECTOR method is called by WRITE-SEQUENCE, this argument is guaranteed to be no greater than end and not greater than the length of vector.

end---a non-negative integer. When a STREAM-WRITE-VECTOR method is called by WRITE-SEQUENCE, this argument is guaranteed to be no less than end and not greater than the length of vector.

Description:

should try to write successive elements of vector to stream, starting at element start (inclusive) and continuing through element end (exclusive.)

[Generic Function]

ccl::stream-device s direction
Returns the OS file descriptor associated with a given lisp stream.

Method Signatures:
ccl::stream-device (s stream) direction => fd
Arguments and Values:

s---a stream.

direction---either :INPUT or :OUTPUT.

fd---a file descriptor, which is a non-negative integer used by the OS to refer to an open file, socket, or similar I/O connection. NIL if there is no file descriptor associated with s in the direction given by direction.

Description:

Returns the file descriptor associated with s in the direction given by direction. It is necessary to specify direction because the input and output file descriptors may be different; the most common case is when one of them has been redirected by the Unix shell.

[Generic Function]

stream-read-ivector stream ivector start-octet max-octets

Description:

Reads up to max-octets octets from stream into ivector, storing them at start-octet. Returns the number of octets actually read.

Arguments:

stream---An input stream. The method defined on BUFFERED-INPUT-STREAMs requires that the size in octets of an instance of the stream's element type is 1.

ivector---Any ivector.

start-octet---A non-negative integer.

max-octets---A non-negative integer. The return value may be less than the value of this parameter if EOF was encountered.

[Generic Function]

stream-write-ivector stream ivector start-octet max-octets

Description:

Writes max-octets octets to stream from ivector, starting at start-octet. Returns max-octets.

Arguments:

stream---An input stream. The method defined on BUFFERED-OUTPUT-STREAMs requires that the size in octets of an instance of the stream's element type is 1.

ivector---Any ivector

start-octet---A non-negative integer.

max-octet---A non-negative integer.

Examples:
;;; Write the contents of a (SIMPLE-ARRAY(UNSIGNED-BYTE 16) 3) 
;;; to a character file stream. Read back the characters.
(let* ((a (make-array 3 
                      :element-type '(unsigned-byte 16)
                      :initial-contents '(26725 27756 28449))))
  (with-open-file (s "junk"
                     :element-type 'character
                     :direction :io
                     :if-does-not-exist :create
                     :if-exists :supersede)
    ;; Write six octets (three elements).
    (stream-write-ivector s a 0 6)
    ;; Rewind, then read a line
    (file-position s 0)
    (read-line s)))

;;; Write a vector of DOUBLE-FLOATs. Note that (to maintain
;;; alignment) there are 4 octets of padding before the 0th 
;;; element of a (VECTOR DOUBLE-FLOAT) on 32-bit platforms.
;;; (Note that (= (- target::misc-dfloat-offset 
;;;                  target::misc-data-offset) 4))
(defun write-double-float-vector
    (stream vector &key (start 0) (end (length vector)))
     (check-type vector (vector double-float))
     (let* ((start-octet (+ (* start 8) 
                            (- target::misc-dfloat-offset
                               target::misc-data-offset)))
	        (num-octets (* 8 (- end start))))
       (stream-write-ivector stream vector start-octet num-octets)))
          


Previous Section Next Section Table of Contents Glossary Index