# Testing framework for q (.qu)

## Overview

qcumber (or quke) is a unit testing framework for q/kdb+ code. The framework offers assertion based testing, benchmarking capabilities, and property based testing.

## Basic usage

\l qcumber.q_
.qu.runTestFile :test.quke
.qu.runTestFolder :tests/


## Writing tests

A test file is created as nested groups of feature, should, bench, and property test blocks.

### File organization

Test file names must end in .quke. Test files are commonly located in a module whose name matches the function's module name, followed by .test. For example, a test file for the function .ab.c would have the name c.quke and be found in the .ab.test module (.ab.test/c.quke).

### Quke syntax overview

Here is a simple test file, containing feature, should, and expect blocks. Most tokens allow for a description to be written to the right of the token to help identify the block and document the test.

feature .ab.c
should exhibit some behaviour
expect some output
1b  / the test case


### Formatting errors

If any formatting errors are detected in a quke file, no tests will be run on that file and an error message will be displayed to the user. Clicking on the error message will open the malformed quke file at the line number where the error was detected. If multiple errors exist, only the first error detected will be displayed to the user.

Many blocks contain q code (q function blocks). If a q function block in a given file cannot compile, qcumber will report an error for that block. Depending on the parent block, an error that occurs when a q function block executes may cause the feature to abort. All errors will be displayed to the user. Every line (other than the first line) in q q function block must have leading whitespace; otherwise the block cannot compile and it will error. This is due to the semantics of q.

### qcumber blocks

The qcumber blocks are described in detail below:

#### Feature

feature blocks form the main test block of a quke file. Each feature block may contain one or more should, bench, property, before, after, before each, or after each blocks. The should, bench, and property blocks contain test cases, while the other four blocks are used to prepare the test environment conditions. A quke file must contain at least one feature block.

name parent block content description allowed abort on error
Feature NA Required:
At least one block

Optional:
One or more should blocks
One or more bench blocks
One or more property blocks
One or more before blocks
One or more after blocks
One or more before each blocks
One or more after each blocks
Yes NA
##### Examples
feature may have a description

before
.ab.num1: 0
after
delete num1 from .ab
before each
.ab.num1: 1
after each
.ab.num1: 0

should may have a description
expect may have a description
.ab.num1 = 1
expect
.ab.num1 = 1

property may have a description
.qch.check .qch.forall[.qch.g.int[]] { x=x }


Multiple features are allowed in an quke file.

feature
should
expect
1b
feature
should
expect
1b


#### Should

should blocks enclose groups of expect blocks.

name parent block content description allowed abort on error
should feature One or more expect blocks yes NA
##### Examples
feature
should
expect
1b


Multiple should blocks are allowed within a feature.

feature .ab.c
should may have a description
expect one
1b
expect two
1 = 1
expect three
any 0010b

should
expect
1b


#### Expect

expect blocks are the assertion based unit test blocks for qcumber. They should return a boolean value, indicating whether or not the test passed, or the result of .qu.compare.

The function .qu.compare accepts an expected and actual value, and can be used in expect blocks to report these values in the test results. It returns 1b if the inputs match, or a dictionary of the actual and expected results if they differ.

The arguments to .qu.compare are stored in the test results table. This can consume significant memory for large values, and performing the comparison in an expect block would be more performant.

.qu.compare replaces the deprecated to match block.

name parent block content description allowed abort on error
expect should Required: q function block Yes No
Optional: (Deprecated) One to match block
##### Examples
feature
should
expect
1b

feature
should
expect may have a description
// This block spans multiple lines,
// and is parsed into a single function
v:0;
v+:1;
v+:1;
v=2;

feature
should
// The following blocks pass
expect
3 ~ count 1 2 3
expect
3
should
// The following test will fail, reporting the actual and expected values
expect
.qu.compare[1; 2]


#### To match

The to match block is deprecated, as unlike .qu.compare it cannot be evaluated in an editor. Use an expect with .qu.compare instead.

to match blocks specify the expected result from an expect block. The test will fail unless both blocks return the same value.

The results from the to match and expect blocks are stored in the test results table. This can consume significant memory for large values, and performing the comparison in an expect block would be more performant.

name parent block content description allowed abort on error
to match expect q function block yes no
##### Examples
feature

should
expect
count 1 2 3
to match
3


#### Bench

The bench block performs benchmarking tests where the runtime of a q function in a behaviour block is compared to the runtime of a baseline block or to a timelimit set in a timelimit block. The bench block may contain behaviour, baseline, timelimit, setup, teardown, tolerance, and replicate blocks.

name parent block content description allowed abort on error
bench feature Required:One behaviour block
One baseline block, one timelimit block, or one of each
If there is a tolerance block, one baseline block is required
Optional:
One or more setup blocks
One or more teardown blocks
One tolerance block
One replicate block
yes NA

When running functions that are relatively close in runtime (for example, til 10000 as a baseline function and til 9000 as a behaviour function), the baseline test will sometimes fail even though the behaviour function should run slightly faster. This occurs due to overhead from memory allocation, and may cause the bench test(s) to fail. There are two ways of dealing with this issue.

1) The user can increase the number of replicates of the run, so that any variation in runtime due to memory allocation will average out. 2) The user can set the number of secondary threads in the workspace to zero. Doing so allows garbage collection to run after each replicate of the bench test. Although this makes every run of the baseline or behaviour q function block slightly slower, it makes runtimes for each function more consistent. This strategy almost always guarantees that the expected functions will run faster. Using both solutions together tends to yield the best results.

The smallest measurable time difference can vary depending on the operating system. The runtimes of baseline and behaviour block functions will be indistinguishable if their runtime is less than the smallest measurable time difference. The smallest measurable time difference is 1 microsecond for most modern operating systems, though for some older Windows versions it is known to be closer to 1 millisecond.

Within a bench test, up to four sub-tests may be run. If any of these tests fail then the entire bench test will be considered to have failed. These tests are detailed in the following table. To find out more about these tests see the blocks listed in the relevant blocks column.

name relevant blocks description
Baseline behaviour baseline Indicates that the behaviour ran slower than the baseline. This test only runs if there is a baseline block and no tolerance block.
Timelimit behaviour timelimit Indicates that the behaviour had a runtime greater than a certain time (in ms). This test only runs if there is a timelimit block.
Upper Tolerance behaviour behaviour tolerance Indicates that the behaviour had a runtime greater than some percentage (the upper tolerance value) of the baseline runtime.
Lower Tolerance behaviour behaviour tolerance Indicates that the behaviour had a runtime greater than some percentage (the lower tolerance value) of the baseline runtime.
This test only runs if there is a tolerance block with two values.
##### Examples
bench
baseline
til 100000000
behaviour
til 1

feature                     // bench blocks must be wrapped in feature blocks
// a bench test must have either one baseline block, one timelimit block, or one of each
bench
baseline
til 100000000
behaviour
til 1
bench                  // a feature may contain several bench blocks
timelimit
100
behaviour
til 1
bench may have a description
timelimit
100
baseline
til 10000000
behaviour
til 1

feature
bench
setup              // multiple setup blocks are allowed
.ab.num1 : 1

setup
.ab.num2 : 1

baseline
til 100000000

behaviour
til 1

teardown           // multiple teardown blocks are allowed
delete num1 from .ab

teardown
delete num2 from .ab

feature
bench
replicate
10             // a single replicate is allowed

tolerance
10             // a single tolerance is allowed

timelimit
10             // a single timelimit is allowed. A timelimit block is required if there is no baseline block.

baseline
til 100000000  // a single baseline is allowed. A baseline block is required if there is no timelimit or if there is a tolerance block.

behaviour          // a single behaviour is required
til 1


#### Behaviour

behaviour blocks contain the q function whose runtime is being benchmarked. In a bench block, the behaviour block runtime is compared to either a timelimit, the runtime of a baseline block, or both. A bench block must have exactly one behaviour block.

name parent block content description allowed abort on error
behaviour bench q function block yes no

Note: behaviour may alternatively be spelled behavior in the quke file.

##### Examples
behaviour
til 20                                  // q function block

feature
bench
timelimit
100                            // timelimit of 100 milliseconds

baseline
til 10000000                   // runtime approximately 10 milliseconds

behaviour may have a description   // behaviour blocks must be wrapped in bench blocks
til 1                          // this will have a runtime within the baseline runtime and within the timelimit, and the test will pass


#### Baseline

baseline blocks contain the q function whose runtime is being compared to a behaviour block. A bench block may have at most one baseline block. A bench block must contain a baseline block if no timelimit block is present or if there is a tolerance block. If no tolerance block is present, the behaviour block must have a runtime less than or equal to the runtime of the baseline block to pass the Baseline test. If a tolerance block is present then the behaviour block must have a runtime within some percentages (set by the tolerance block) of the baseline block runtime to pass the Upper Tolerance and/or Lower Tolerance tests.

name parent block content description allowed abort on error
baseline bench q function block yes no
##### Examples
baseline
til 200                                 // q function block

feature
bench
baseline may have a description    // baseline blocks must be wrapped in bench blocks
til 10000000                   // runtime approximately 10 milliseconds

behaviour
til 1                          // this will have a runtime within the baseline runtime, and the test will pass


#### Timelimit

timelimit blocks set the maximum runtime (in milliseconds) for a behaviour block. A bench block may have at most one timelimit block. If no baseline block is present a bench block must have a timelimit block. The behaviour block must have a runtime less than or equal to the timelimit to pass the Timelimit test (see above).

name parent block content description allowed abort on error
timelimit bench One long datatype no NA
##### Examples
timelimit
10                  // timelimit of 10 milliseconds

feature
bench
timelimit      // timelimit blocks must be wrapped in bench blocks
100        // timelimit of 10 milliseconds

behaviour
til 1      // this will have a runtime less than the timelimit, and the test will pass


#### Tolerance

tolerance blocks set tolerance limits for the benchmark tests. The tolerance block may contain one or two numerical values. If one value is present it is the upper tolerance value. If two values are present the larger one is the upper tolerance value and the smaller one is the lower tolerance value.

Two tolerance tests exist: the Upper Tolerance test and the Lower Tolerance test.

In the upper tolerance test, the behaviour must execute with a runtime less than or equal to some percentage (the upper tolerance value) of the baseline runtime to pass the Upper Tolerance test.

In the lower tolerance test, the behaviour must execute with a runtime greater than or equal to some percentage (the lower tolerance value) of the baseline runtime to pass the Lower Tolerance test. The lower tolerance test only runs if there is a lower tolerance value.

A bench block that contains a tolerance block must also have a baseline block.

name parent block content description allowed abort on error
tolerance bench One or two float or long datatypes no NA
##### Examples
tolerance
90                  // a tolerance with only an upper tolerance value of 90%
// the behaviour must have a runtime less than 90% of the baseline runtime to pass the upper tolerance test
// the lower tolerance test will not run

tolerance
10 90               // a tolerance block with both upper and lower tolerance values of 90% and 10% respectively
// the behaviour must have a runtime less than 90% of the baseline runtime to pass the upper tolerance test
// the behaviour must have a runtime greater than 10% of the baseline runtime to pass the lower tolerance test

tolerance
10.0 120.0          // a tolerance block with both the upper and lower tolerance values as floats
// the upper tolerance value can be greater than 100%

feature
bench
baseline         // if a tolerance block is present there must be a baseline block
til 10000000

behaviour
til 5000000

tolerance        // tolerance blocks must be wrapped in bench blocks
10 90         // the behaviour will have a runtime within the given percentages of the baseline runtime


#### Replicate

replicate blocks set the number of replicates to be performed on the test functions of the bench block. The average runtimes will be used in the benchmarking tests. The number of replicates defaults to 1 if no replicate block is present. The number of replicates must be greater than zero.

name parent block content description allowed abort on error
replicate bench One long datatype that is greater than zero no NA
##### Examples
replicate
10                  // 10 replicates

 feature
bench
replicate     // replicate blocks must be wrapped in bench blocks
10        // 10 replicates. The behaviour will be executed 10 times, and its average runtime will be found.

timelimit
100

behaviour
til 1

feature
bench
replicate
10         // 10 replicates. The behaviour and baseline will each execute 10 times, and their average runtimes will be found and compared.

baseline
10000000

behaviour
til 1


#### Setup

setup blocks execute at the beginning of a bench block. They are used for test preparation by, for example, setting global variables to be used in the tests. Errors thrown in setup blocks cause the feature to abort.

name parent block content description allowed abort on error
setup bench q function block no yes
##### Examples
setup
a::1                    // q function block

feature
bench
setup                // setup blocks must be wrapped in bench blocks
.ab.num1 : 1      // the setup runs at the beginning of the bench block

baseline
til 10000000

behaviour
til 1

feature
bench
setup
.ab.num1 : 1      // multiple setups are allowed

setup
.ab.num1 : 1

baseline
til 10000000

behaviour
til 1


#### Teardown

teardown blocks execute at the beginning of a bench block. They are used for test cleanup. For example, deleting global variables from the workspace. Errors thrown in teardown blocks cause the feature to abort.

name parent block content description allowed abort on error
teardown bench q function block no yes
##### Examples
teardown
a::0                            // q function block

feature
bench
setup
.ab.num1 : 1

baseline
til 10000000

behaviour
til 1

teardown                     // setup blocks must be wrapped in bench blocks
delete num1 from .ab   // the teardown runs at the end of the bench block

feature
bench
setup
.ab.num1 : 1;
.ab.num2 : 2;

baseline
til 10000000

behaviour
til 1

teardown
delete num1 from .ab   // multiple teardowns are allowed

teardown
delete num2 from .ab


#### Property

property blocks allow for property based testing using the QuickCheck framework provided by the .qch module. This allows multiple tests to be run with randomly generated input parameters. Although QuickCheck tests can be run within an expect block, the property block has the benefit of displaying any counter examples to the user.

A property block must contain q that returns the results of a valid QuickCheck expression. The valid QuickCheck expression must contain the function .qch.check, followed by one of the seven .qch.forall functions. See the .qch module for more information. This expression returns a dictionary indicating whether the test passed or failed, any counter-examples to the test, and any errors that occurred while running the test.

A handy .qch function to use is .qch.with.times. This allows the user to set the number of samples to test in a given property block. An example of .qch.with.times, along with other property blocks, can be seen below.

name parent block content description allowed abort on error
property feature valid QuickCheck expression yes no
##### Examples
property
// a valid QuickCheck expression, with the function .qch.check followed by a .qch.forall function
.qch.check
.qch.forall[.qch.g.int[]]{
x=x
}

feature
property another property                           // property blocks must be wrapped in feature blocks
// another valid QuickCheck expression, containing .qch.check followed by the .qch.forall2 function
.qch.check
.qch.forall2[.qch.g.int[];.qch.g.int[]]{
all .axq.isInt each x, y
}

feature
property
a::1;                                          // additional q code is allowed prior to the QuickCheck expression in the property block
.qch.check
.qch.forall[.qch.g.int[]]{
1 in a,x
}

// By default, QuickCheck does 100 samples to look for errors
// The function .qch.with.times can be used to alter the number of samples.
feature
property
.qch.check
.qch.with.times[10]             // Set the number of samples to 10
.qch.forall[.qch.g.int[]]{
1 = x
}


#### Before

before blocks execute after any skip if blocks in a feature. They are used for test preparation. For example, setting global variables to be used in the tests. Errors thrown in before blocks cause the feature to abort.

name parent block content description allowed abort on error
before feature q function block no yes
##### Examples
before
.ab.num1 : 1            // q function bloc

feature
before                  // before blocks must be wrapped in feature blocks
.ab.num1 : 1        // before blocks execute at the start of their parent feature

feature
before
.ab.num1 : 2

feature                     // multiple before blocks are allowed in a feature
before
.ab.num1 : 1

before
.ab.num2 : 2


#### Before each

before each blocks execute prior to each should and each property block in a feature block that is not skipped. They are used for test preparation. For example, setting global variables to be used in the tests. Errors thrown in before each blocks cause the feature to abort.

name parent block content description allowed abort on error
before each feature q function block no yes
##### Examples
before each
.ab.num1 : 1            // q function block

feature
before each             // before each blocks must be wrapped in feature blocks
.ab.num1 : 1

before each             // multiple before each blocks are allowed in a feature
.ab.num2 : 2

feature
before each
.ab.num1 : 1       // this before each block executes prior to both of the should blocks

should
expect
show .ab.num1;         // will display 1
.ab.num1 :2;
1b

expect
show .ab.num1;         // will display 2
.ab.num1 :2;
1b

should
expect
show .ab.num1;         // will display 1
.ab.num1 : 2;
1b


#### After

after blocks execute at the end of a feature block. They are used for test cleanup. For example, deleting any global variables defined by the test.

name parent block content description allowed abort on error
after feature q function block no no
##### Examples
after
.ab.num1 : 1            // q function block

feature
after                   // after blocks must be wrapped in feature blocks
.ab.num1: 1

after                   // multiple after blocks are allowed in a feature
.ab.num2 : 2


feature                     // after blocks execute at the end of their parent feature
after
.ab.num1: 1

feature
after
.ab.num2 : 2


#### After each

after each blocks execute after each should and each property block in a feature block that is not skipped. They are used for test cleanup. For example, deleting any global variables defined by the test. Errors thrown in after each blocks cause the feature to abort.

name parent block content description allowed abort on error
after each feature q function block no yes
##### Examples
after each
.ab.num1 : 1                // q function block

feature
after each                  // after each blocks must be wrapped in feature blocks
.ab.num1 : 1

after each                  // multiple after each blocks are allowed in a feature
.ab.num2 : 2

feature
before
.ab.num1 : 1
after each
.ab.num1 : 2                 // this after each block executes after every should block

should
expect
show .ab.num1;            // will display 1
1b

expect
show .ab.num1;            // will display 1
1b

should
expect
show .ab.num1;            // will display 2
1b


#### Skip if

skip if blocks execute at the beginning of a feature. They are used to validate conditions and determine whether or not the tests should be run. If the block returns boolean true (1b), then the feature is skipped, and all of its blocks are reported as skipped blocks. If the skip if block returns boolean false (0b), then feature is executed. If a non-boolean value is returned, or if an error is thrown, the feature aborts.

name parent block content description allowed abort on error
skip if feature q function block no yes
##### Examples
   skip if
.z.o like "l*"      // q function block, will skip if os is linux

feature                    // multiple "skip if" blocks are allowed in a feature
skip if
.z.o like "m*"
skip if
w64 ~ .z.o

// "skip if" blocks execute at the start of their parent feature, and determine whether the rest of the feature is tested
feature
skip if
.z.o like "m*"
feature
skip if
w64 ~ .z.o


### Skipping tests

It is possible to skip individual test blocks by prepending an x to the block name. The test blocks that may be skipped are feature, should, expect, bench, and property. For example, to skip an expect block, replace the expect with xexpect. Skipping a test causes it to be completely ignored during the test run. No parsing errors will be reported for these blocks. Skipping a block may cause a parse error in any enclosing blocks if that block was expected in the enclosing block. Any skipped tests will be reported in the output of the test runner.

It is also possible to skip tests dynamically using a skip if block. See above for details.

## Running qcumber

Two functions are provided to run test files programmatically: .qu.runTestFile and .qu.runTestFolder. The arguments to both functions are a string or symbol which represents the path to the file or folder respectively. Both relative and absolute paths are accepted. The function .qu.runTestFolder will recursively find and run all test files within a folder.

.qu.runTestFolder "folder"
.qu.runTestFile   "folder/file"

.qu.runTestFolder :folder
.qu.runTestFile   :folder/file


If generic null is passed to .qu.runTestFolder then it will recursively run all tests in the current directory.

.qu.runTestFolder[]


The results dictionary has the structure:

key value
allTestResults A table containing the results from all tests.
allFailedTestResults A table containing the results from all failed tests.
allSkippedTestResults A table containing the tests that were skipped.
parseErrorList A list of strings reporting every quke file with parse errors.

The result table reported by the keys allTestResults, allFailedTestResults, and allSkippedTestResults have the following schema:

column type description
namespace symbol The folder path.
fileName symbol The name of the file.
feature symbol The name of the feature.
block symbol The block type.
description string A description of the block.
expectations string If the block is should, then this will be the description associated with individual expect block in the should block.
line long The line number in the file.
success boolean Whether or not the test was successful.
result any The result returned by the test. Differs depending on the test block type.
error string Any errors found.
aborted boolean Whether or not the test was aborted.
skipped boolean Whether or not the test was skipped.
parseError boolean Whether or not the test file had a parse error.
start timestamp The start time.
time timespan The time taken to run the test.

In the above table, the result column may have different structures depending on the block type. If the block is should, then the result column will contain a dictionary with the following structure:

key value type value
expect any If the test failed, the result of the expect block, or .qu.compare. Null otherwise.
toMatch any If the test failed, the result of .qu.compare, or 1b if .qu.compare was not used. Null otherwise.
expectError string Any errors thrown by the expect block.
toMatchError string Any errors thrown by a deprecated to match block.

If the block is bench, then the result column will contain a dictionary with the following structure:

key value type value
baseline string The description of the baseline block.
behaviour string The description of the behaviour block.
baselineError string Any errors thrown by the baseline block.
behaviourError string Any errors thrown by the behaviour block.
passedBaseline boolean Whether or not the Baseline test passed.
passedTimelimit boolean Whether or not the Timelimit test passed.
passedLowerTolerance boolean Whether or not the Lower Tolerance test passed.
passedUpperTolerance boolean Whether or not the Upper Tolerance test passed.
timeBehaviour float The runtime of the behaviour block in ms.
timeBaseline float The runtime of the behaviour block in ms. Null if there was no baseline block.
timelimit long The set timelimit in ms. Null if there was no timelimit block.

If the block is property, then the result column will contain a dictionary with the following structure:

key value type value
output string The raw output from the property block. This should normally correspond to a QuickCheck dictionary (see .qch for more details). If the property block was improperly formatted then it may have a different data structure.
formatError boolean Whether or not the property block was properly formatted. If this is true (1b) then the block did not correspond to a valid QuickCheck expression, and you cannot query into the output column to investigate the expected QuickCheck dictionary.
failed any The counter example. If possible this will be the shrunk value from .qch.

If the block is none of the above, the then the result column holds generic null.

To load qcumber into a q process, refer to the Running Libraries section of the Notes & TroubleShooting guide.

## Mocking functions

Qcumber provides utilities to mock user functions. This can facilitate testing by allowing users to replace the implementation of functions with their mocked functions that allow verifying that the correct arguments are supplied, or by controlling the return type to easily test the calling functions.

To mock functions, either use .qu.stub.single to mock a single function or .qu.stub.mock to mock a set of functions. .qu.stub.single takes the name of the function to mock and the replacement function as arguments. The argument to .qu.stub.mock is a list, where each item is a list consisting of the name of the function to replace and the replacement function.

To restore all of the mocked functions to their original versions call .qu.stub.restoreAll. To restore only a subset of functions, call .qu.stub.restore. The argument to .qu.stub.restore is a list of the functions to restore. If a given function was mocked multiple times then .qu.stub.restore will only restore the function to the version prior to its current version.

## Debugging tests

To debug a failing test, errors within tests can be propagated rather than trapped by calling

.qu.setBreakOnErrors 1b


Calling the same function with 0b resets this behaviour.

## Examples

### File examples

Examples test files are presented below. These show the main types of qcumber blocks:

• A feature block, illustrating the terminology used in the documentations.
feature                 // qcumber token: designates a feature block
should              // qcumber token: designates a should block
expect          // qcumber token: designates a expect block
v:0;        // |
v+:1;       // |
v+:1;       // | q function block: this block spans multiple lines but is parsed into a single function
v+:1;       // |
v = 3;      // |

expect          // qcumber token: designates a second expect block
1b          // q function block: a second q function, though this one spans only one line

expect          // qcumber token: designates a second expect block
[a:8;       // |
b:8;       // | q function block: a q function block may optionally be enclosed by a single pair of square brackets
a=b]       // |

• Expect block tests.
// expect blocks are used to run assertion tests
feature
should this is a should
expect this is an expect
1b

expect this is an expect
1 = 1

• A bench block test.
// bench blocks are used to perform runtime tests
feature
bench this is a bench
setup
.ab.num1 : 1

timelimit
100

tolerance
10 100

replicate
1000

baseline this is a baseline
til 10000

behaviour this is a behaviour
til 5000

teardown
delete num1 from .ab

• A property block test.
// property blocks are used to perform tests on sets of randomly generated values
feature
property this is a property
.qch.check
.qch.forall[.qch.g.int[]]{
x=x
}

• A quke file showing all types of blocks, as well as multiple features. In additions, comments are shown after tokens that allow descriptions.
// A feature block containing all of the possible qcumber blocks
feature this is a feature                       // descriptions are allowed after feature blocks
before
.ab.num1 : 0

after
delete num1 from .ab

before each
.ab.num1 : 1

after each
.ab.num1 : 0

should this is a should                     // descriptions are allowed after should blocks
expect this is an expect                // descriptions are allowed after expect blocks
.ab.num1 = 1

expect this is another expect           // descriptions are allowed after expect blocks
.ab.num1 = 1

bench this is a bench                       // descriptions are allowed after bench blocks
setup
.ab.num2 : 1

timelimit
100

tolerance
10 100

replicate
1000

baseline this is a baseline            // descriptions are allowed after baseline blocks
til 10000

behaviour this is a behaviour          // descriptions are allowed after behaviour blocks
til 5000

teardown
delete num2 from .ab

property this is a property                 // descriptions are allowed after property
.qch.check
.qch.forall[.qch.g.int[]]{
x=x
}


### Failing test examples

Examples of the expected outputs for failing tests are shown below:

• An example failing expect test, along with the failed test output, is shown below.

Consider a test file .ab.test/expectExample.quke with the following content:

feature
should
expect
0b


This test will fail because the expect block returns a value other than boolean true (in this case, it returns boolean false). When the user runs the test, it will display the following failed test message:

1 of 1 test failed

Failed Tests : 1
feature
should
expect  (.ab.test/expectExample.quke.quke:3)
Expected Result: 1b
Actual Result: 0b


Clicking on the tag (.ab.test/expectExample.quke:3) will open the failed test file on the line with the block that failed (in this case, line 3). An example of a failing bench test, along with the failed test output, is shown below.

Consider a test file .ab.test/benchExample.quke with the following content:

feature
bench
baseline
til 1

behaviour
til 10000000


When the user runs the test, it will display a failed test message similar to the following:

1 of 1 test failed

Failed Tests : 1
feature
bench  (.ab.test/benchExample.quke:2)
Behaviour Runtime: 8.85 ms
Baseline Runtime: 0.001 ms
Baseline test: Failed


In this case, the Baseline test failed, which indicates that the behaviour ran slower than the baseline. The runtimes for the behaviour and baseline as well as the timelimit (if present) are reported to the user.

An example of a failing property test, along with the failed test output, is shown below.

Consider a test file .ab.test/propertyExample.quke with the following content:

feature
property
.qch.check
.qch.forall[.qch.g.int[]]{
.axq.isFloat x
}


This test will fail because no integer is a float.

When the user runs the test, it will display a failed test message similar to the following:

1 of 1 test failed

Failed Tests : 1
feature
property  (.ab.test/benchExample.quke:2)
Counter Example: ,0i


In this case, the counter example that broke the test is displayed to the user. If an error occurred in the QuickCheck expression, it will also be displayed to the user.

### Skipped test examples

An example of a skipped expect test, along with the test output is shown below. Consider a test file .ab.test/skippedExample.quke with the following content:

feature skippedExample.quke
should exhibit a behaviour
xexpect a specific result
0b

expect a specific result
0b


This test has two failing expects, but only the latter is reported since the first test was skipped. When the user runs the test, it will display the following failed test message:

1 of 2 tests failed
1 test skipped

Failed Tests : 1
feature skippedExample
should exhibit a behaviour
expect a specific result (.ab.test/skippedExample.quke:6)
Expected Result: 1b
Actual Result: 0b
Skipped Tests : 1


Clicking on the tag (.ab.test/skippedExample.quke:6) will open the failed test file on line 6 where the block failed. Both the "Failed Tests" and "Skipped Tests" tags can be clicked to open an close drop down menus that describe the failed and skipped tests in further detail. To view the skipped tests, click on the arrow next to the "Skipped Tests" (not visible in this document).

## Block summary

name description parent block content description allowed abort on error
feature Contains all other blocks NA Required:
At least one child block
Optional:
One or more before blocks
One or more after blocks
One or more before each blocks
One or more after each blocks
One or more should blocks
One or more bench blocks
One or more property blocks
yes NA
should Wrapper block to contain expects feature One or more expect blocks yes NA
expect Assertion test block should Required:
q function block
Optional:
(Deprecated) one to match block
yes no
to match (Deprecated) Expected result from preceding expect block expect q function block yes no
bench Benchmarking test block feature Required:
One behaviour block
One baseline block, one timelimit block, or one of each
If there is a tolerance block, one baseline block is required

Optional:
One or more setup blocks
One or more teardown blocks
One tolerance block
One replicate block
yes NA
behaviour Code whose runtime is compared to a timelimit or the baseline runtime bench q function block yes no
baseline Baseline code whose runtime is compared to behaviour function bench q function block yes no
timelimit Timelimit for behaviour code bench One long datatype no NA
tolerance Upper and lower tolerance percentages for baseline/behaviour runtime comparison bench One or two float or long datatypes no NA
replicate Number of times to measure runtime of behaviour and baseline code bench One long datatype that is greater than zero no NA
setup Runs code at start of a bench bench q function block no yes
teardown Runs code at end of a bench bench q function block no yes
property Property test block feature valid QuickCheck expression yes no
before Runs code at the start of a feature feature q function block no yes
before each Runs code prior to each should in a feature feature q function block no yes
after Runs code at the end of a feature feature q function block no no
after each Runs code after each should in a feature feature q function block no yes
skip if Runs code at the start of a feature to decide if the rest of the feature is run feature q function block no yes

## .qu.compare

Compares two values such that the actual vs. expected values can be included in qcumber results

Parameters:

Name Type Description
actual any The output being tested
expected any The reference value

Returns:

Type Description
boolean | dict (expect: any; toMatch: any) 1b if the inputs match, or a dictionary of the actual and expected results if they differ

## .qu.runTestFile

Runs a single test file with default settings

Parameter:

Name Type Description
path string | symbol The folder path

Returns:

Type Description
.qu.results See .qu/README.md for more information about the structure of the results table given different block types.

Example:

 .qu.runTestFile "file.quke"


## .qu.runTestFileWithSettings

Runs a single test file

Parameters:

Name Type Description
path string | symbol The folder path
sts .qu.ty.userSettings | null The qcumber settings

Returns:

Type Description
.qu.results See Running qcumber for more information about the structure of the results table given different block types.

Example:

 .qu.runTestFile "file.quke"


## .qu.runTestFolder

Runs all tests in a folder with default settings

Parameter:

Name Type Description
path string | symbol | null The folder path. Null is replaced with "." namespace

Returns:

Type Description
.qu.results See Running qcumber for more information about the structure of the results table given different block types.

Example:

 .qu.runTestFolder "folder"


## .qu.runTestFolderWithSettings

Runs all tests in a folder

Parameters:

Name Type Description
path string | symbol | null The folder path. Null is replaced with "." namespace
sts .qu.ty.userSettings | null The qcumber settings

Returns:

Type Description
.qu.results See Running qcumber for more information about the structure of the results table given different block types.

Example:

 .qu.runTestFolder "folder"
`

## .qu.setBreakOnErrors

Allows errors from unit tests to propagate, facilitate debugging

Parameter:

Name Type Description
x boolean 1b to not trap errors in unit tests

Returns:

Type Description
null

## .qu.settings.default

the default settings for qcumber