| Previous Section | Next Section | Table of Contents | Glossary | Index |
In Clozure CL 1.4 and later, code coverage provides information about which paths through generated code have been executed and which haven't. For each source form, it can report one of three possible outcomes:
Not covered: this form was never entered.
Partly covered: This form was entered, and some parts were executed and some weren't.
Fully covered: Every bit of code generated from this form was executed.
While the information gathered for coverage of generated code is complete and precise, the mapping back to source forms is of necessity heuristic, and depends a great deal on the behavior of macros and the path of the source forms through compiler transforms. Source information is not recorded for variables, which further limits the source mapping. In practice, there is often enough information scattered about a partially covered function to figure out which logical path through the code was taken and which wasn't. If that doesn't work, you can try disassembling to see which parts of the compiled code were not executed: in the disassembled code there will be references to #<CODE-NOTE [xxx] ...> where xxx is NIL if the code that follows was never executed and non-NIL if it was.
Sometimes the situation can be improved by modifying macros to try to preserve more of the input forms, rather than destructuring and rebuilding them.
Because the code coverage information is associated with compiled functions, code coverage information is not available for load-time toplevel expressions. You can work around this by creating a function and calling it. I.e. instead of
(progn (do-this) (setq that ...) ...))
do:
(defun init-this-and-that () (do-this) (setq that ...) ...) (init-this-and-that)
Then you can see the coverage information in the definition of
init-this-and-that.
In order to gather code coverage information, you first have to
recompile all your code to include code coverage
instrumentation. Compiling files will generate code coverage
instrumentation if CCL:*COMPILE-CODE-COVERAGE*
is true:
(setq ccl:*compile-code-coverage* t) (recompile-all-your-files)
The compilation process will be many times slower than normal, and the fasl files will be many times bigger.
When you execute functions loaded from instrumented fasl files, they
will record coverage information every time they are executed.
You can examine that information by calling ccl:report-coverage
or ccl:coverage-statistics.
While recording coverage, you can collect incremental coverage deltas between any two points in time. You might do this while running a test suite, to record the coverage for each test, for example:
(ccl:reset-incremental-coverage)
(loop with coverage = (make-hash-table)
for test in (tests-to-run)
do (run-test test)
do (setf (gethash test coverage) (ccl:get-incremental-coverage))
finally (return coverage))
creates a hash table mapping a test to a representation of all coverage recorded while running the
test. This hash table can then be passed to ccl:report-coverage, ccl:incremental-coverage-svn-matches
or ccl:incremental-coverage-source-matches.
The following functions can be used to manage the coverage data:
report-coverage output-file &key
(tags nil) (external-format :default) (statistics t) (html t)
output-file--- Pathname for the output index file.
html--- If non-nil (the default), this will generate an HTML report, consisting of an index file in output-file and, in the same directory, one html file for each instrumented source file that has been loaded in the current session.
tags--- If non-nil, this should be a hash table mapping arbitrary keys (tags) to incremental coverage deltas. The HTML report will show a list of tags, and allow selection of an arbitrary subset of them to show the coloring and statistics for coverage by that subset.
external-format--- Controls the external format of the html files.
statistics--- If non-nil (the default), a comma-separated file is generated with the summary of statistics. You can specify a filename for the statistics argument, otherwise "statistics.csv" is created in the directory of output-file. See documentation of coverage-statistics below for a description of the values in the statistics file.
Restores the coverage data previously saved with
ccl:save-coverage-in-file, for the set of instrumented fasls
that were loaded both at save and restore time. I.e. coverage
info is only restored for files that have been loaded in this
session. For example if in a previous session you had loaded
"foo.lx86fsl" and then saved the coverage info, in this session
you must load the same "foo.lx86fsl" before calling
restore-coverage-from-file in order to retrieve the stored
coverage info for "foo". Equivalent to (ccl:restore-coverage
(ccl:read-coverage-from-file pathname)).
Returns a sequence of ccl:coverage-statistics objects, one for each
source file, containing the same information as that written to
the statistics file by ccl:report-coverage. The following
accessors are defined for ccl:coverage-statistics objects:
the name of the source file corresponding to this information
the total number of expressions
the number of source expressions that have been entered (i.e. at least partially covered)
the number of source expressions that were fully covered
the number of conditionals with one branch taken and one not taken
the total number of code forms. A code form is an expression in the final stage of compilation, after all macroexpansion and compiler transforms and simplification
the number of code forms that have been entered
the total number of functions
the number of functions that were fully covered
the number of functions that were partly covered
the number of functions never entered
Returns the delta of coverage since the last reset of incremental coverage.
If reset is true (the default), it also resets incremental coverage
now, so that the next call to get-incremental-coverage will return
the delta from this point.
Incremental coverage deltas are represented differently than the full coverage snapshots
returned by functions such as ccl:get-coverage. Incremental
coverage uses an abbreviated format
and is missing some of the information in a full snapshot, and therefore cannot be passed to
functions documented to accept a snapshot, only to functions
specifically documented to accept incremental coverage deltas.
collection--- A hash table mapping arbitrary keys to incremental coverage deltas, or a sequence of incremental coverage deltas.
sources--- A list of pathnames and/or source-notes, the latter representing a range within a file.
Given a hash table collection whose values are incremental coverage
deltas, return a list of all keys corresponding to those deltas that intersect any region
in sources.
For example if the deltas represent tests, then the returned value is a list of all tests that cover some part of the source regions.
collection can also be a sequence of deltas, in which case a subsequence
of matching deltas is returned. In particular you can test whether any particular delta
intersects the sources by passing it in as a single-element list.
incremental-coverage-svn-matches collection &key (directory (current-directory)) (revision :base)
collection--- A hash table mapping arbitrary keys to incremental coverage deltas, or a sequence of incremental coverage deltas.
directory--- The pathname of a subversion working directory.
revision---
The revision to compare to the working directory, an integer or another
value whose printed representation is suitable for passing as the
--revision argument
to svn.
Given a hash table collection whose values are incremental coverage
deltas, return a list of all keys corresponding to those deltas that intersect any changed
source in directory since revision revision in subversion.
For example if the deltas represent tests, then the returned value is a list of all tests that might be affected by the changes.
collection can also be a sequence of deltas, in which case a subsequence
of matching deltas is returned. In particular you can test whether any particular delta
is affected by the changes by passing it in as a single-element list.
The output of ccl:report-coverage consists of formatted source code, with coverage indicated by coloring. Four colors are used: dark green for forms that compiled to code in which every single instruction was executed, light green for forms that have been entered but weren't totally covered, red for forms that were never entered, and the page background color for toplevel forms that weren't instrumented.
The source coloring is applied from outside in. So for example if you have
(outer-form ... (inner-form ...) ...)
first the whole outer form is painted with whatever color expresses the outer form coverage, and then the inner form color is replaced with whatever color expresses the inner form coverage. One consequence of this approach is that every part of the outer form that is not specifically inside some executable inner form will have the outer form's coverage color. If the syntax of outer form involves some non-executable forms, or forms that do not have coverage info of their own for whatever reason, then they will just inherit the color of the outer form, because they don't get repainted with a color of their own.
One case in which this approach can be confusing is in the case of symbols. As noted in the Limitations section, coverage information is not recorded for variables; hence the coloring of a variable does not convey information about whether the variable was evaluated or not -- that information is not available, and the variable just inherits the color of the form that contains it.
| Previous Section | Next Section | Table of Contents | Glossary | Index |