The class CodeCoverage

Overview

CodeCoverage watches selected parts of the workspace and records which lines of code are actually executed. It is built on ⎕PROFILE.

Typically it is used while running a test framework such as Tester2, but measuring coverage by simply running an application can be useful as well, especially for legacy code.

Note: CodeCoverage releases are published as Tatin packages, see there.

Note: CodeCoverage requires the Unicode edition of Dyalog APL.

Getting started

The following assumes that there is an application in #.Foo we are interested in:

      C←⎕NEW #.CodeCoverage (,⊂'#.Foo')
      C.ignore←'#.Foo.TestCases,#.Foo.Samples'
      C.filename←'./Foo-Coverage'
      C.Start ⍬
      #.Foo.Run ⍬
      C.Stop ⍬
      C.Finalise ⍬
./Foo-Coverage.profile
      ⎕←#.CodeCoverage.ProcessDataAndCreateReport C.filename
./Foo-Coverage.html

Step by step:

• The argument of ⎕NEW is a comma-separated list of the names of the namespaces and/or classes to be watched. Names that are not fully qualified are looked up in the namespace ⎕NEW is called from. The name of a variable holding a reference may be used as well; it is resolved to the name of its target.

• ignore names anything you are not interested in; see The ignore property.

• filename is the base name for all files CodeCoverage creates; see The filename property. If you do not set it, a name in the temp directory is used.

• Start switches ⎕PROFILE on, Stop switches it off and collects what was gathered. You may stop and start again as often as you like: ⎕PROFILE is expensive, so it is worth excluding the parts you don't care about, for example the test framework itself.

• Finalise analyses the code of everything that was watched, works out which lines were executed and which could have been, and writes the result to disk. It returns the name of the .profile file, which is why the example shows a line of output.

• ProcessDataAndCreateReport aggregates the data collected by all runs and creates the HTML report. It returns the name of the HTML file.

By default the report lists, for every function and operator, the fully qualified name, the lines that were not executed, the coverage percentage and the total number of testable lines. Passing a left argument of 1 (see The verbose option) adds the body of every partially covered function and operator to the report, with the lines that were not executed printed in bold and marked with a right-pointing arrow.

What counts as a line of code

These lines are not counted as executable, and therefore neither help nor hurt the coverage percentage:

• Empty lines and lines that carry nothing but a comment
• The header line of a traditional function or operator
• The first line of a dfn or dop, but only when it consists of nothing but Name←{; if it carries code, it counts
• The closing } of a dfn or dop
• Lines that consist of nothing but :Else, any flavour of :End, :Access, :Field, :Attribute, :Implements, :Signature or :Section

A function or operator that is left with no testable line at all, a one-line dfn for example, is judged by its first line instead: it counts as covered once it has been called.

An example report

A real-world example, taken from two runs of the test suite of the FilesAndDirs package:

The test suite was run twice, once on Windows and once on Linux, and the two sets of data were aggregated into this single report. That matters here because FilesAndDirs executes different lines of code depending on the operating system in a considerable number of cases; see Running the code more than once.

The four functions that are not covered at all are worth a look in either case:

• If the report came from running a test suite, they are simply untested.

• If the report came from running the application, it may be worth checking whether they are obsolete.

Ideally, test cases should cover 100% of the code, but that is not always feasible. However, for functions with a significant number of executable lines, a coverage of less than 50% seems unacceptable.

Running the code more than once

Data from several runs can be aggregated into a single report. There are two typical reasons for doing so:

1. An application with a server part and a client part that share one or more namespaces. Only after running both test suites can a meaningful report be compiled.

2. A multi-platform application that must run on all supported platforms before its coverage can be judged.

To do this, run the code again with filename pointing at the file the earlier run created. Finalise then appends to that file rather than creating a new one:

      C←⎕NEW #.CodeCoverage (,⊂'#.Foo')
      C.filename←'./Foo-Coverage'      ⍝ Already exists: this run will be added to it
      C.Start ⍬
      #.Foo.RunClientTests ⍬
      C.Stop ⍬
      C.Finalise ⍬
./Foo-Coverage.profile
      ⎕←1 #.CodeCoverage.ProcessDataAndCreateReport C.filename
./Foo-Coverage.html

Create the report only when the last run is done. ProcessDataAndCreateReport may be called on a file at any point, and as often as you like, but it can only report on the runs performed so far.

The code must not change between runs

All runs must be performed on exactly the same code: coverage is recorded as line numbers, so adding, deleting or changing a line invalidates the data collected by earlier runs.

To make such a mistake detectable, Finalise saves the canonical representation of everything it watched in a file with the extension .cr, and checks it on every subsequent run. Two messages may be printed to the session:

• Current definition of function "…" does not match the one captured earlier - do not trust the results

  The code did change between runs. The report is worthless; delete the .profile and .cr files (see DeleteFiles) and start again.

• Found new function "…" that was not captured in previous runs - do not trust the results!

  A function or operator turned up in a namespace that an earlier run watched as well, which normally means the same thing.

  A namespace that no earlier run watched does not trigger this message: watching different namespaces in different runs is exactly what the two scenarios above are about.

What is stored where

.profile is an ordinary component file:

Component	Contents
1	Documentation of the file's own layout
2	The namespaces and classes that were watched
3	The names that are to be ignored
4	One row per run: timestamp, APL version, ⎕WA in MB
5-9	Reserved
10	The aggregated data; empty until ProcessData has been called
11 …	One component per run, appended by Finalise

Each of those per-run components is a matrix with four columns: the name of the function or operator, the line numbers that were executed, the coverage percentage, and the total number of testable lines.

ProcessData aggregates all of them and writes the result into component 10.

More details

The filename property

filename is a base name, specified with a path but without an extension; anything after the last . is removed. Up to four files are derived from it:

Extension	Contents
.profile	The collected coverage data
.cr	The canonical representation of all watched code, used to detect changes between runs
.html	The report created by CreateReport
.md	The summary created by CreateSummary

By default filename points at a temporary directory; a directory that does not exist yet is created by Finalise. DeleteFiles removes the .profile and .cr files again; with a left argument of 1 the reports, .html and .md, go as well.

The ignore property

ignore is a character vector, empty by default, holding a comma-separated list of fully qualified names that are to be left out of the report. Functions, operators, namespaces and classes may all be named; naming a namespace ignores everything in it, including its sub-namespaces. This is decided by the names alone, so it also works when the report is compiled in a workspace where those objects do not exist.

Note that ignore is an instance property, so it must be set before Finalise is called.

The verbose option

verbose is the optional left argument of both CreateReport and ProcessDataAndCreateReport, and defaults to 0. This means that the body of a partially covered function or operator is not listed; you get the fully qualified name, which lines were not executed, the coverage percentage and the total number of lines.

That is usually what you want for a first report: there will be quite a few functions that are not yet covered by the tests, and you will probably add tests for some of them, and exclude others you have good reason not to care about — those that are too trivial to be worth covering, for example. See The ignore property.

Once that is done, run the tests again and generate the report with verbose←1: now the details are worth looking at.

Creating a summary

ProcessDataAndCreateSummary writes a small Markdown file (extension .md) rather than a full HTML report: how many functions and operators were watched, how many runs were performed and when, the overall percentage of covered code, and how many functions and operators are covered 100%. It is meant to be used as a job summary in a GitHub workflow.

The methods

Shared:

Method	Purpose
ProcessData	Aggregate and massage the data collected by all runs
CreateReport	Create the HTML report from the aggregated data
ProcessDataAndCreateReport	Both of the above in one go
CreateSummary	Create the Markdown summary from the aggregated data
ProcessDataAndCreateSummary	Both of the above in one go
DeleteFiles	Delete the files belonging to a report
NamesInScript	The names of all functions and operators within a class or interface script
Version, History	Version number and change log

CreateReport and CreateSummary rely on the data having been aggregated, so ProcessData must be called first. Using the combined methods is simpler; call the individual ones only when you want both a report and a summary without aggregating twice.

Instance:

Name	Purpose
Start, Stop	Switch ⎕PROFILE on and off
Finalise	Analyse the watched code and write the collected data to disk
watch	The namespaces and classes being watched; read-only, established by the constructor
ignore	What is to be left out of the report
filename	The base name of all files created
data	The raw data collected so far; read-only

Limitations

• Scripts nested inside other scripts (classes within classes, scripted namespaces within scripted namespaces, or a mixture of them) are not supported: the inner objects are simply ignored. Ordinary namespaces are processed at any depth, including any scripts they contain.

• At the time of writing, Dyalog provides an ⌶ for retrieving the code of functions and operators — including private ones — from scripts, but nothing that helps to analyse a script in order to extract the names of all of them, so CodeCoverage does the analysing itself.

  For this to work, CodeCoverage needs the code to be properly formatted. For example, the number of leading spaces is what distinguishes a ∇ used for a recursive call in a dfn from a ∇ used as the opening or closing tag of a traditional function.

• Dfns, named or unnamed, that are defined locally within a dfn or dop are not recognised as such, but only as part of the outer function.

• Named dfns defined locally within a traditional function or operator are completely ignored. The lines they occupy are therefore reported, wrongly, as not covered.

  The reason is that ⎕PROFILE does not provide the necessary information.

• This statement:

  :If a≡1 ⋄ MyFns 1 ⋄ MyOtherFns 2 ⋄ :EndIf

  is reported as "covered" as soon as the interpreter reaches the line, although clearly only one of the two function calls can have been executed.

  Again, this is a ⎕PROFILE limitation that cannot be overcome. There are already a number of good reasons not to use ⋄, and this is just another one.