Skip to content

Testing framework for q

Overview

qcumber (or quke) is a unit testing framework for q/kdb+ code. The framework offers assertion based testing, benchmarking capabilities, and property based testing.

Basic usage

\l qcumber.q_
.qu.runTestFile `:test.quke
.qu.runTestFolder `:tests/

Writing tests

A test file is created as nested groups of feature, should, bench, and property test blocks.

File organization

Test file names must end in .quke. Test files are commonly located in a module whose name matches the function's module name, followed by .test. For example, a test file for the function .ab.c would have the name c.quke and be found in the .ab.test module (.ab.test/c.quke).

Quke syntax overview

Here is a simple test file, containing feature, should, and expect blocks. Most tokens allow for a description to be written to the right of the token to help identify the block and document the test.

feature .ab.c
    should exhibit some behaviour
        expect some output
            1b  / the test case

Formatting errors

If any formatting errors are detected in a quke file, no tests will be run on that file and an error message will be displayed to the user. Clicking on the error message will open the malformed quke file at the line number where the error was detected. If multiple errors exist, only the first error detected will be displayed to the user.

Many blocks contain q code (q function blocks). If a q function block in a given file cannot compile, the entire quke file will abort and an error message will be displayed to the user. Depending on the parent block, an error that occurs when a q function block executes may cause the feature to abort. All errors will be displayed to the user.

qcumber blocks

The qcumber blocks are described in detail below:

Feature

feature blocks form the main test block of a quke file. Each feature block may contain one or more should, bench, property, before, after, before each, or after each blocks. The should, bench, and property blocks contain test cases, while the other four blocks are used to prepare the test environment conditions. A quke file must contain at least one feature block.

Name Parent Block Content Description Allowed Abort On Error
Feature NA Required:
At least one block

Optional:
One or more should blocks
One or more bench blocks
One or more property blocks
One or more before blocks
One or more after blocks
One or more before each blocks
One or more after each blocks
Yes NA
Examples
feature may have a description

    before
        .ab.num1: 0
    after
        delete num1 from `.ab
    before each
        .ab.num1: 1
    after each
        .ab.num1: 0

    should may have a description
        expect may have a description
            .ab.num1 = 1
        expect
            .ab.num1 = 1

    property may have a description
        .qch.check .qch.forall[.qch.g.int[]] { x=x }

Multiple features are allowed in an quke file.

feature
    should
        expect
            1b
feature
    should
        expect
            1b

Should

should blocks enclose groups of expect blocks.

Name Parent block Content Description allowed Abort on error
should feature One or more expect blocks yes NA
Examples
feature
    should
        expect
            1b

Multiple should blocks are allowed within a feature.

feature .ab.c
    should may have a description
        expect one
            1b
        expect two
            1 = 1
        expect three
            any 0010b

    should
        expect
            1b

Expect

expect blocks are the assertion based unit test blocks for qcumber. An expect block may be used alone or be associated with a to match block directly following the expect block.

If no to match block is present, then the q function within an expect block should return a boolean value indicating whether or not the test passed. Any value other than boolean true (1b) will be reported as a failed test.

If a to match block follows the expect block, the q functions within both the expect and to match block should return the same result for the test to pass. If the results do not match then the test will fail.

Name Parent block Content Description allowed Abort on error
expect should Required: q function block Yes No
Optional: One to match block
Examples
feature
    should
        expect
            1b
feature
    should
        expect may have a description
            v:0;                    / q function block: this
            v+:1;                   / block spans multiple
            v+:1;                   / lines but is parsed
            v=2;                    / into a single function
feature
    should
        expect
            [a:8;          / q function block: a q function
             b:8;          / block may optionally be enclosed
             a=b]          / by a single pair of square brackets
feature
    // The following two should blocks are equivalent
    // Using a to match block to specify the expected return
    // from an expect block often leads to cleaner syntax
    should
        expect
            3 ~ count 1 2 3
    should
        expect
            count 1 2 3
        to match
            3

To match

to match blocks are used to specify the expected result from an expect block. By default, the test will fail if the expect block returns anything other than boolean true (1b). With the to match block it is possible to specify an alternative test result. This result will be compared with the return value of the expect block, and if they are exactly equal the test will pass. This block can help clean up the structure of tests.

Name Parent block Content Description allowed Abort on error
to match expect q function block yes no
Examples
feature
    // these two should blocks are equivalent
    // however, the second can often be cleaner syntax if complex data structures are involved
    should
        expect
            3 ~ count 1 2 3
    should
        expect
            count 1 2 3
        to match
            3

Bench

The bench block performs benchmarking tests where the runtime of a q function in a behaviour block is compared to the runtime of a baseline block or to a timelimit set in a timelimit block. The bench block may contain behaviour, baseline, timelimit, setup, teardown, tolerance, and replicate blocks.

Name Parent block Content Description allowed Abort on error
bench feature Required:One behaviour block
One baseline block, one timelimit block, or one of each
If there is a tolerance block, one baseline block is required
Optional:
One or more setup blocks
One or more teardown blocks
One tolerance block
One replicate block
yes NA

When running functions that are relatively close in runtime (for example, til 10000 as a baseline function and til 9000 as a behaviour function), the baseline test will sometimes fail even though the behaviour function should run slightly faster. This occurs due to overhead from memory allocation, and may cause the bench test(s) to fail. There are two ways of dealing with this issue.

1) The user can increase the number of replicates of the run, so that any variation in runtime due to memory allocation will average out. 2) The user can set the number of slaves in the workspace to zero. Doing so allows garbage collection to run after each replicate of the bench test. Although this makes every run of the baseline or behaviour q function block slightly slower, it makes runtimes for each function more consistent. This strategy almost always guarantees that the expected functions will run faster. Using both solutions together tends to yield the best results.

The smallest measurable time difference can vary depending on the operating system. The runtimes of baseline and behaviour block functions will be indistinguishable if their runtime is less than the smallest measurable time difference. The smallest measurable time difference is 1 microsecond for most modern operating systems, though for some older Windows versions it is known to be closer to 1 millisecond.

Within a bench test, up to four sub-tests may be run. If any of these tests fail then the entire bench test will be considered to have failed. These tests are detailed in the following table. To find out more about these tests see the blocks listed in the relevant blocks column.

Name Relevant blocks Description
Baseline behaviour baseline Indicates that the behaviour ran slower than the baseline. This test only runs if there is a baseline block and no tolerance block.
Timelimit behaviour timelimit Indicates that the behaviour had a runtime greater than a certain time (in ms). This test only runs if there is a timelimit block.
Upper Tolerance behaviour behaviour tolerance Indicates that the behaviour had a runtime greater than some percentage (the upper tolerance value) of the baseline runtime.
Lower Tolerance behaviour behaviour tolerance Indicates that the behaviour had a runtime greater than some percentage (the lower tolerance value) of the baseline runtime.
This test only runs if there is a tolerance block with two values.
Examples
bench
    baseline
        til 100000000
    behaviour
        til 1
feature                     // bench blocks must be wrapped in feature blocks
                            // a bench test must have either one baseline block, one timelimit block, or one of each
     bench
         baseline
             til 100000000
         behaviour
             til 1
     bench                  // a feature may contain several bench blocks
         timelimit
             100
         behaviour
             til 1
     bench may have a description
         timelimit
             100
         baseline
             til 10000000
         behaviour
             til 1
feature
     bench
         setup              // multiple setup blocks are allowed
             .ab.num1 : 1

         setup
             .ab.num2 : 1

         baseline
             til 100000000

         behaviour
             til 1

         teardown           // multiple teardown blocks are allowed
             delete num1 from `.ab

         teardown
             delete num2 from `.ab
feature
     bench
         replicate
             10             // a single replicate is allowed

         tolerance
             10             // a single tolerance is allowed

         timelimit
             10             // a single timelimit is allowed. A timelimit block is required if there is no baseline block.

         baseline
             til 100000000  // a single baseline is allowed. A baseline block is required if there is no timelimit or if there is a tolerance block.

         behaviour          // a single behaviour is required
             til 1

Behaviour

behaviour blocks contain the q function whose runtime is being benchmarked. In a bench block, the behaviour block runtime is compared to either a timelimit, the runtime of a baseline block, or both. A bench block must have exactly one behaviour block.

Name Parent block Content Description allowed Abort on error
behaviour bench q function block yes no

Note: behaviour may alternatively be spelled behavior in the quke file.

Examples
behaviour
    til 20                                  // q function block
feature
     bench
         timelimit
             100                            // timelimit of 10 milliseconds

         baseline
             til 10000000                   // runtime approximately 10 milliseconds

         behaviour may have a description   // behaviour blocks must be wrapped in bench blocks
             til 1                          // this will have a runtime within the baseline runtime and within the timelimit, and the test will pass

Baseline

baseline blocks contain the q function whose runtime is being compared to a behaviour block. A bench block may have at most one baseline block. A bench block must contain a baseline block if no timelimit block is present or if there is a tolerance block. If no tolerance block is present, the behaviour block must have a runtime less than or equal to the runtime of the baseline block to pass the Baseline test. If a tolerance block is present then the behaviour block must have a runtime within some percentages (set by the tolerance block) of the baseline block runtime to pass the Upper Tolerance and/or Lower Tolerance tests.

Name Parent block Content Description allowed Abort on error
baseline bench q function block yes no
Examples
baseline
    til 200                                 // q function block
feature
     bench
         baseline may have a description    // baseline blocks must be wrapped in bench blocks
             til 10000000                   // runtime approximately 10 milliseconds

         behaviour
             til 1                          // this will have a runtime within the baseline runtime, and the test will pass

Timelimit

timelimit blocks set the maximum runtime (in milliseconds) for a behaviour block. A bench block may have at most one timelimit block. If no baseline block is present a bench block must have a timelimit block. The behaviour block must have a runtime less than or equal to the timelimit to pass the Timelimit test (see above).

Name Parent block Content Description allowed Abort on error
timelimit bench One long datatype no NA
Examples
timelimit
    10                  // timelimit of 10 milliseconds
feature
     bench
         timelimit      // timelimit blocks must be wrapped in bench blocks
             100        // timelimit of 10 milliseconds

         behaviour
             til 1      // this will have a runtime less than the timelimit, and the test will pass

Tolerance

tolerance blocks set tolerance limits for the benchmark tests. The tolerance block may contain one or two numerical values. If one value is present it is the upper tolerance value. If two values are present the larger one is the upper tolerance value and the smaller one is the lower tolerance value.

Two tolerance tests exist: the Upper Tolerance test and the Lower Tolerance test.

In the upper tolerance test, the behaviour must execute with a runtime less than or equal to some percentage (the upper tolerance value) of the baseline runtime to pass the Upper Tolerance test.

In the lower tolerance test, the behaviour must execute with a runtime greater than or equal to some percentage (the lower tolerance value) of the baseline runtime to pass the Lower Tolerance test. The lower tolerance test only runs if there is a lower tolerance value.

A bench block that contains a tolerance block must also have a baseline block.

Name Parent block Content Description allowed Abort on error
tolerance bench One or two float or long datatypes no NA
Examples
tolerance
    90                  // a tolerance with only an upper tolerance value of 90%
                        // the behaviour must have a runtime less than 90% of the baseline runtime to pass the upper tolerance test
                        // the lower tolerance test will not run
tolerance
    10 90               // a tolerance block with both upper and lower tolerance values of 90% and 10% respectively
                        // the behaviour must have a runtime less than 90% of the baseline runtime to pass the upper tolerance test
                        // the behaviour must have a runtime greater than 10% of the baseline runtime to pass the lower tolerance test
tolerance
    10.0 120.0          // a tolerance block with both the upper and lower tolerance values as floats
                        // the upper tolerance value can be greater than 100%
feature
    bench
       baseline         // if a tolerance block is present there must be a baseline block
         til 10000000

       behaviour
         til 5000000

       tolerance        // tolerance blocks must be wrapped in bench blocks
          10 90         // the behaviour will have a runtime within the given percentages of the baseline runtime

Replicate

replicate blocks set the number of replicates to be performed on the test functions of the bench block. The average runtimes will be used in the benchmarking tests. The number of replicates defaults to 1 if no replicate block is present. The number of replicates must be greater than zero.

Name Parent block Content Description allowed Abort on error
replicate bench One long datatype that is greater than zero no NA
Examples
replicate
    10                  // 10 replicates
 feature
      bench
          replicate     // replicate blocks must be wrapped in bench blocks
              10        // 10 replicates. The behaviour will be executed 10 times, and its average runtime will be found.

          timelimit
              100

          behaviour
              til 1
feature
     bench
         replicate
             10         // 10 replicates. The behaviour and baseline will each execute 10 times, and their average runtimes will be found and compared.

         baseline
             10000000

         behaviour
             til 1

Setup

setup blocks execute at the beginning of a bench block. They are used for test preparation by, for example, setting global variables to be used in the tests. Errors thrown in setup blocks cause the feature to abort.

Name Parent block Content Description allowed Abort on error
setup bench q function block no yes
Examples
setup
    a::1                    // q function block
feature
    bench
       setup                // setup blocks must be wrapped in bench blocks
          .ab.num1 : 1      // the setup runs at the beginning of the bench block

       baseline
         til 10000000

       behaviour
         til 1
feature
    bench
       setup
          .ab.num1 : 1      // multiple setups are allowed

       setup
          .ab.num1 : 1

       baseline
         til 10000000

       behaviour
         til 1

Teardown

teardown blocks execute at the beginning of a bench block. They are used for test cleanup. For example, deleting global variables from the workspace. Errors thrown in teardown blocks cause the feature to abort.

Name Parent block Content Description allowed Abort on error
teardown bench q function block no yes
Examples
teardown
    a::0                            // q function block
feature
    bench
       setup
            .ab.num1 : 1

       baseline
            til 10000000

       behaviour
            til 1

       teardown                     // setup blocks must be wrapped in bench blocks
            delete num1 from `.ab   // the teardown runs at the end of the bench block
feature
    bench
       setup
            .ab.num1 : 1;
            .ab.num2 : 2;

       baseline
            til 10000000

       behaviour
            til 1

       teardown
            delete num1 from `.ab   // multiple teardowns are allowed

       teardown
            delete num2 from `.ab

Property

property blocks allow for property based testing using the QuickCheq framework provided by the .qch module. This allows multiple tests to be run with randomly generated input parameters. Although QuickCheq tests can be run within an expect block, the property block has the benefit of displaying any counter examples to the user.

A property block must contain q that returns the results of a valid QuickCheq expression. The valid QuickCheq expression must contain the function .qch.check, followed by one of the seven .qch.forall functions. See the .qch module for more information. This expression returns a dictionary indicating whether the test passed or failed, any counter-examples to the test, and any errors that occurred while running the test.

A handy .qch function to use is .qch.with.times. This allows the user to set the number of samples to test in a given property block. An example of .qch.with.times, along with other property blocks, can be seen below.

Name Parent block Content Description allowed Abort on error
property feature valid QuickCheq expression yes no
Examples
property
    // a valid QuickCheq expression, with the function .qch.check followed by a .qch.forall function
    .qch.check
        .qch.forall[.qch.g.int[]]{
            x=x
        }
feature
    property another property                           // property blocks must be wrapped in feature blocks
         // another valid QuickCheq expression, containing .qch.check followed by the .qch.forall2 function
        .qch.check
            .qch.forall2[.qch.g.int[];.qch.g.int[]]{
                all .axq.isInt each x, y
            }
feature
    property
         a::1;                                          // additional q code is allowed prior to the QuickCheq expression in the property block
         .qch.check
             .qch.forall[.qch.g.int[]]{
                1 in a,x
             }
// By default, QuickCheq does 100 samples to look for errors
// The function `.qch.with.times` can be used to alter the number of samples.
feature
    property
         .qch.check
             .qch.with.times[10]             // Set the number of samples to 10
                 .qch.forall[.qch.g.int[]]{
                     1 = x
                 }

Before

before blocks execute at the beginning of a feature block. They are used for test preparation. For example, setting global variables to be used in the tests. Errors thrown in before blocks cause the feature to abort.

Name Parent block Content Description allowed Abort on error
before feature q function block no yes
Examples
before
    .ab.num1 : 1            // q function bloc
feature
    before                  // before blocks must be wrapped in feature blocks
        .ab.num1 : 1        // before blocks execute at the start of their parent feature

feature
    before
        .ab.num1 : 2
feature                     // multiple before blocks are allowed in a feature
    before
        .ab.num1 : 1

    before
        .ab.num2 : 2

Before each

before each blocks execute prior to each should and each property block in a feature block. They are used for test preparation. For example, setting global variables to be used in the tests. Errors thrown in before each blocks cause the feature to abort.

Name Parent block Content Description allowed Abort on error
before each feature q function block no yes
Examples
before each
    .ab.num1 : 1            // q function block
feature
    before each             // before each blocks must be wrapped in feature blocks
        .ab.num1 : 1

    before each             // multiple before each blocks are allowed in a feature
        .ab.num2 : 2
feature
    before each
         .ab.num1 : 1       // this before each block executes prior to both of the should blocks

    should
        expect
            show .ab.num1;         // will display 1
            .ab.num1 :2;
            1b

        expect
            show .ab.num1;         // will display 2
            .ab.num1 :2;
            1b

    should
        expect
            show .ab.num1;         // will display 1
            .ab.num1 : 2;
            1b

After

after blocks execute at the end of a feature block. They are used for test cleanup. For example, deleting any global variables defined by the test.

Name Parent block Content Description allowed Abort on error
after feature q function block no no
Examples
after
    .ab.num1 : 1            // q function block
feature
    after                   // after blocks must be wrapped in feature blocks
        .ab.num1: 1

    after                   // multiple after blocks are allowed in a feature
        .ab.num2 : 2

feature                     // after blocks execute at the end of their parent feature
    after
        .ab.num1: 1

feature
    after
        .ab.num2 : 2

After each

after each blocks execute after each should and each property block in a feature block. They are used for test cleanup. For example, deleting any global variables defined by the test. Errors thrown in after each blocks cause the feature to abort.

Name Parent block Content Description allowed Abort on error
after each feature q function block no yes
Examples
after each
    .ab.num1 : 1                // q function block
feature
    after each                  // after each blocks must be wrapped in feature blocks
        .ab.num1 : 1

    after each                  // multiple after each blocks are allowed in a feature
        .ab.num2 : 2
feature
     before
         .ab.num1 : 1
     after each
          .ab.num1 : 2                 // this after each block executes after every should block

     should
         expect
             show .ab.num1;            // will display 1
             1b

         expect
             show .ab.num1;            // will display 1
             1b

     should
         expect
             show .ab.num1;            // will display 2
             1b

Skipping tests

It is possible to skip individual test blocks by prepending an x to the block name. The test blocks that may be skipped are feature, should, expect, bench, and property. For example, to skip an expect block, replace the expect with xexpect. Skipping a test causes it to be completely ignored during the test run. No parsing errors will be reported for these blocks. Skipping a block may cause a parse error in any enclosing blocks if that block was expected in the enclosing block. Any skipped tests will be reported in the output of the test runner.

Running qcumber

Two functions are provided to run test files programmatically: .qu.runTestFile and .qu.runTestFolder. The arguments to both functions are a string or symbol which represents the path to the file or folder respectively. Both relative and absolute paths are accepted. The function .qu.runTestFolder will recursively find and run all test files within a folder.

.qu.runTestFolder "folder"
.qu.runTestFile   "folder/file"

.qu.runTestFolder `:folder
.qu.runTestFile   `:folder/file

If generic null is passed to .qu.runTestFolder then it will recursively run all tests in the current directory.

.qu.runTestFolder[]

The results dictionary has the structure:

Key Value
allTestResults A table containing the results from all tests.
allFailedTestResults A table containing the results from all failed tests.
allSkippedTestResults A table containing the tests that were skipped.
parseErrorList A list of strings reporting every quke file with parse errors.

The result table reported by the keys allTestResults, allFailedTestResults, and allSkippedTestResults have the following schema:

Column Type Description
namespace symbol The folder path.
fileName symbol The name of the file.
feature symbol The name of the feature.
block symbol The block type.
description string A description of the block.
expectations string If the block is `Should, then this will be the description associated with individual expect block in the should block.
line long The line number in the file.
success boolean Whether or not the test was successful.
result any The result returned by the test. Differs depending on the test block type.
error string Any errors found.
aborted boolean Whether or not the test was aborted.
skipped boolean Whether or not the test was skipped.
parseError boolean Whether or not the test file had a parse error.
start timestamp The start time.
time timespan The time taken to run the test.

In the above table, the result column may have different structures depending on the block type. If the block is `should, then the result column will contain a dictionary with the following structure:

Key Value type Value
expect any The result returned from the expect block.
toMatch any The result returned from the to match block, or 1b if no such block exists.
expectError string Any errors thrown by the expect block.
toMatchError string Any errors thrown by the to match block.

If the block is `bench, then the result column will contain a dictionary with the following structure:

Key Value type Value
baseline string The description of the baseline block.
behaviour string The description of the behaviour block.
baselineError string Any errors thrown by the baseline block.
behaviourError string Any errors thrown by the behaviour block.
passedBaseline boolean Whether or not the Baseline test passed.
passedTimelimit boolean Whether or not the Timelimit test passed.
passedLowerTolerance boolean Whether or not the Lower Tolerance test passed.
passedUpperTolerance boolean Whether or not the Upper Tolerance test passed.
timeBehaviour float The runtime of the behaviour block in ms.
timeBaseline float The runtime of the behaviour block in ms. Null if there was no baseline block.
timelimit long The set timelimit in ms. Null if there was no timelimit block.

If the block is `property, then the result column will contain a dictionary with the following structure:

Key Value type Value
output string The raw output from the property block. This should normally correspond to a QuickCheq dictionary (see .qch for more details). If the property block was improperly formatted then it may have a different data structure.
formatError boolean Whether or not the property block was properly formatted. If this is true (1b) then the block did not correspond to a valid QuickCheq expression, and you cannot query into the output column to investigate the expected QuickCheq dictionary.
failed any The counter example. If possible this will be the shrunk value from .qch.

If the block is none of the above, the then the result column holds generic null.

To load qcumber into a q process, refer to the Running Libraries section of the Notes & TroubleShooting guide.

Examples

File examples

Examples test files are presented below. These show the main types of qcumber blocks:

  • A feature block, illustrating the terminology used in the documentations.
feature                 // qcumber token: designates a feature block
    should              // qcumber token: designates a should block
        expect          // qcumber token: designates a expect block
            v:0;        // |
            v+:1;       // |
            v+:1;       // | q function block: this block spans multiple lines but is parsed into a single function
            v+:1;       // |
            v = 3;      // |

        expect          // qcumber token: designates a second expect block
            1b          // q function block: a second q function, though this one spans only one line

        expect          // qcumber token: designates a second expect block
            [a:8;       // |
             b:8;       // | q function block: a q function block may optionally be enclosed by a single pair of square brackets
             a=b]       // |
  • Expect block tests.
// expect blocks are used to run assertion tests
feature
    should this is a should
        expect this is an expect
            1b

        expect this is an expect
            1 = 1
  • A bench block test.
// bench blocks are used to perform runtime tests
feature
    bench this is a bench
        setup
            .ab.num1 : 1

        timelimit
            100

        tolerance
            10 100

        replicate
            1000

        baseline this is a baseline
            til 10000

        behaviour this is a behaviour
            til 5000

        teardown
            delete num1 from `.ab
  • A property block test.
// property blocks are used to perform tests on sets of randomly generated values
feature
    property this is a property
        .qch.check
            .qch.forall[.qch.g.int[]]{
                x=x
            }
  • A quke file showing all types of blocks, as well as multiple features. In additions, comments are shown after tokens that allow descriptions.
// A feature block containing all of the possible qcumber blocks
feature this is a feature                       // descriptions are allowed after feature blocks
    before
        .ab.num1 : 0

    after
        delete num1 from `.ab

    before each
        .ab.num1 : 1

    after each
        .ab.num1 : 0

    should this is a should                     // descriptions are allowed after should blocks
        expect this is an expect                // descriptions are allowed after expect blocks
            .ab.num1 = 1

        expect this is another expect           // descriptions are allowed after expect blocks
            .ab.num1 = 1

    bench this is a bench                       // descriptions are allowed after bench blocks
        setup
            .ab.num2 : 1

        timelimit
            100

        tolerance
            10 100

        replicate
            1000

        baseline this is a baseline            // descriptions are allowed after baseline blocks
            til 10000

        behaviour this is a behaviour          // descriptions are allowed after behaviour blocks
            til 5000

        teardown
                delete num2 from `.ab

    property this is a property                 // descriptions are allowed after property
        .qch.check
            .qch.forall[.qch.g.int[]]{
                x=x
            }

Failing test examples

Examples of the expected outputs for failing tests are shown below:

  • An example failing expect test, along with the failed test output, is shown below.

Consider a test file .ab.test/expectExample.quke with the following content:

feature
    should
        expect
            0b

This test will fail because the expect block returns a value other than boolean true (in this case, it returns boolean false). When the user runs the test, it will display the following failed test message:

1 of 1 test failed

Failed Tests : 1
feature
    should
        expect  (.ab.test/expectExample.quke.quke:3)
            Expected Result: 1b
            Actual Result: 0b

Clicking on the tag (.ab.test/expectExample.quke:3) will open the failed test file on the line with the block that failed (in this case, line 3). An example of a failing bench test, along with the failed test output, is shown below.

Consider a test file .ab.test/benchExample.quke with the following content:

feature
    bench
        baseline
            til 1

        behaviour
            til 10000000

When the user runs the test, it will display a failed test message similar to the following:

1 of 1 test failed

Failed Tests : 1
feature
    bench  (.ab.test/benchExample.quke:2)
        Behaviour Runtime: 8.85 ms
        Baseline Runtime: 0.001 ms
        Baseline test: Failed

In this case, the Baseline test failed, which indicates that the behaviour ran slower than the baseline. The runtimes for the behaviour and baseline as well as the timelimit (if present) are reported to the user.

An example of a failing property test, along with the failed test output, is shown below.

Consider a test file .ab.test/propertyExample.quke with the following content:

feature
   property
       .qch.check
           .qch.forall[.qch.g.int[]]{
               .axq.isFloat x
           }

This test will fail because no integer is a float.

When the user runs the test, it will display a failed test message similar to the following:

1 of 1 test failed

Failed Tests : 1
feature
    property  (.ab.test/benchExample.quke:2)
        Counter Example: ,0i

In this case, the counter example that broke the test is displayed to the user. If an error occurred in the QuickCheq expression, it will also be displayed to the user.

Skipped test examples

An example of a skipped expect test, along with the test output is shown below. Consider a test file .ab.test/skippedExample.quke with the following content:

feature skippedExample.quke
    should exhibit a behaviour
        xexpect a specific result
            0b

        expect a specific result
            0b

This test has two failing expects, but only the latter is reported since the first test was skipped. When the user runs the test, it will display the following failed test message:

1 of 2 tests failed
1 test skipped

Failed Tests : 1
feature skippedExample
    should exhibit a behaviour
        expect a specific result (.ab.test/skippedExample.quke:6)
            Expected Result: 1b
            Actual Result: 0b
Skipped Tests : 1

Clicking on the tag (.ab.test/skippedExample.quke:6) will open the failed test file on line 6 where the block failed. Both the "Failed Tests" and "Skipped Tests" tags can be clicked to open an close drop down menus that describe the failed and skipped tests in further detail. To view the skipped tests, click on the arrow next to the "Skipped Tests" (not visible in this document).

Block summary

Name Description Parent block Content Description allowed Abort on error
feature Contains all other blocks NA Required:
At least one child block
Optional:
One or more before blocks
One or more after blocks
One or more before each blocks
One or more after each blocks
One or more should blocks
One or more bench blocks
One or more property blocks
yes NA
should Wrapper block to contain expects feature One or more expect blocks yes NA
expect Assertion test block should Required:
q function block
Optional:
one to match block
yes no
to match Expected result from preceding expect block expect q function block yes no
bench Benchmarking test block feature Required:
One behaviour block
One baseline block, one timelimit block, or one of each
If there is a tolerance block, one baseline block is required

Optional:
One or more setup blocks
One or more teardown blocks
One tolerance block
One replicate block
yes NA
behaviour Code whose runtime is compared to a timelimit or the baseline runtime bench q function block yes no
baseline Baseline code whose runtime is compared to behaviour function bench q function block yes no
timelimit Timelimit for behaviour code bench One long datatype no NA
tolerance Upper and lower tolerance percentages for baseline/behaviour runtime comparison bench One or two float or long datatypes no NA
replicate Number of times to measure runtime of behaviour and baseline code bench One long datatype that is greater than zero no NA
setup Runs code at start of a bench bench q function block no yes
teardown Runs code at end of a bench bench q function block no yes
property Property test block feature valid QuickCheq expression yes no
before Runs code at the start of a feature feature q function block no yes
before each Runs code prior to each should in a feature feature q function block no yes
after Runs code at the end of a feature feature q function block no no
after each Runs code after each should in a feature feature q function block no yes

.qu.runTestFile

Runs a single test file with default settings

Parameter(s):

Name Type Description
path string | symbol The folder path

Returns:

Name Type Description
#.qu.results See .qu/README.md for more information about the structure of the results table given different block types.

Example:

 .qu.runTestFile "file.quke"

.qu.runTestFileWithSettings

Runs a single test file

Parameter(s):

Name Type Description
path string | symbol The folder path
sts #.qu.ty.userSettings | null The qcumber settings

Returns:

Name Type Description
#.qu.results See Running qcumber for more information about the structure of the results table given different block types.

Example:

 .qu.runTestFile "file.quke"

.qu.runTestFolder

Runs all tests in a folder with default settings

Parameter(s):

Name Type Description
path string | symbol | null The folder path. Null is replaced with "." namespace

Returns:

Name Type Description
#.qu.results See Running qcumber for more information about the structure of the results table given different block types.

Example:

 .qu.runTestFolder "folder"

.qu.runTestFolderWithSettings

Runs all tests in a folder

Parameter(s):

Name Type Description
path string | symbol | null The folder path. Null is replaced with "." namespace
sts #.qu.ty.userSettings | null The qcumber settings

Returns:

Name Type Description
#.qu.results See Running qcumber for more information about the structure of the results table given different block types.

Example:

 .qu.runTestFolder "folder"

.qu.settings.default

the default settings for qcumber