Skip to content

qCumber

qCumber provides numerous unit-testing features. The primary tests offered are assertion testing, benchmark testing, and property based testing.

Unit tests are stored in files with the quke (.quke) extension. Unit tests consist of a top-level feature block. feature blocks contain nested should and expect blocks as the primary assertion-based test.

feature blocks

contain all other test blocks, along with a single-line descriptions of an application component, such as a function’s name.

should blocks

contain expect blocks, along with a single-line descriptions, typically of a use case that a component fulfills.

expect blocks

contain the q code that implements a unit test, along with a single-line description of the test itself.

feature Count
    should return the count of its input
        expect count on an atom to return 1
            1 ~ count `a
        expect count of 1 drop list to return n - 1
            n:1 + first 1?10;
            l:1 _ n?`8;
            count[l] ~ n - 1

expect blocks contain q code that is executed as a q function. The final statement in the code block is returned by this function and compared to an expected value. In the example above, this value is boolean true (1b). The test passes if the values match.

An expect block may optionally be followed by a to match block. The to match block allows the user to specify the value that is expected to be returned from the expect block. This can allow for cleaner syntax when a function being tested returns a complex data structure. Like the expect block, the to match block contains q code that is executed as a q function. The returned value is compared to the value returned from the expect block, and if the values match then the test passes.

feature Count
    should test flip operator
        expect flip to produce a table
            l : til 6;
            flip `a`b!(l;reverse l)
        to match
            ([] a: 0 1 2 3 4 5; b: 5 4 3 2 1 0)

In addition to should and expect blocks for assertion based testing, qCumber provides bench blocks for benchmark testing and property blocks for property-based testing. These are described in more detail below.

Running tests and error reporting

Tests are executed by right-clicking on an artifact in the artifact tree, clicking the Code > Run Tests. Depending on the artifact type this will yield different results.

Running the tests on a

  • repository will run all of the .quke files under a repository
  • module will run all of the .quke files under a module
  • quke file will run the tests contained in that file

    testSearchpane.png

Multiple files, modules, and repositories may be tested simultaneously by multi-selecting artifacts and following the above instructions. It is also possible to test an individual file by right clicking an open editor window and clicking Test.

Tests can be associated with a particular function (See Conventions, namespaces, and contexts below). This allows you to run the tests for a function by clicking on the function.

Output

The output for qCumber tests reports the tests that failed, as well as any malformed test files and any skipped tests.

The failed tests, malformed files, and skipped tests are nested in the output as dropdown lists that can be opened by clicking the arrow.

failedTest.png

The failed tests section will display the test blocks that failed, the file name and line number where the failed test occurred, and, depending on the type of test, details as to the reason that the test failed. These could be failed test conditions or errors in the q code. Clicking the line number where the error occurred will open the failed test file at the test that failed.

The malformed files section displays a list of formatting errors as well as the file name and line number where the error occurred. These can also be clicked to open the location of the error.

malformedTest.png

Skipped tests show only the identity and location of the skipped tests (see below).

Skipping tests

qCumber supports ignoring tests by prepending an x to the front of a given test block. qCumber supports skipping feature, should, expect, bench, and property blocks.

For example, in the following test the first expect block is skipped. In the editor the skipped section is grayed out in the editor.

feature count
    should return the count of its input
        xexpect to skip this test
            1 ~ count `a
        expect correct count of list of length 2
            2 ~ count `a`b

The output from this test would report the skipped test. To view the skipped tests in the output, click the arrow to the right of the Skipped Tests header.

skippedTest.png

Before and after blocks

before, after, before each, and after each are optional blocks that can be written immediately after a feature block:

before blocks

will execute arbitrary code before running the first should, bench, or property block for a feature

after blocks

will execute arbitrary code after running the first should, bench, or property block for a feature

before each blocks

will execute arbitrary code before running each should, bench, or property block for a feature

after each blocks

will execute arbitrary code after running each should, bench, or property block for a feature

feature demoBeforeAndAfter
    before
        // Generate 50 random numbers between 0-50 before a test runs
        `randomNumbers set 50?50i;
    after
        // Delete the global variable we created in the before block
        delete randomNumbers from (system "d")

    should use data created in our before block
        expect the variable to exist
         `randomNumbers in key system "d"
        expect its count to be 50
          50 ~ count randomNumbers
        expect it to be integers
          `ints ~ typeOf randomNumbers

feature demoBeforeEach
    before each
        // Grow a list before each test runs
        $[`randomList in key system "d";
            // If the list exists append to it
            randomList,:1?`8;
            // Else create a new list
            `randomList set 1?`8
            ];
    after each
        delete randomList from (system "d")

    should showcase that our list is growing
        expect the count to be 1
            1 ~ count randomList
    should showcase that our list is still growing
        expect the count to be 2
            2 ~ count randomList
    should showcase that our list is still growing again
        expect the count to be 3
            3 ~ count randomList

Conventions, namespaces, and contexts

By convention, we name a quke file after the function that it tests. We keep quke files in a separate module whose name is moduleName.test. For example, the tests for .demo would be placed in a .demo.test module, and the test for the function .demo.foo would be placed in .demo.test/foo.quke. You can also put the test in the module containing the function to test (i.e. .demo/foo.quke) though this is discouraged.

namespaces.png

If you follow these conventions then you can run the tests in .demo.test by right-clicking on .demo, assuming that both modules are present in your workspace. You can also run the tests for the function by right-clicking on the function.

Any module that has a leading dot in its name defines a namespace or context. Tests run under the context of the module that the test file belongs to. For example, .demo.test would run its tests in the .demo context because the namespace of the module is .demo. This means that tests can use relative function names, making renaming tests and modules simpler.

Benchmark blocks

qCumber provides a facility for benchmarking and comparing code runtime. A simple example is the following:

feature benchmark
    bench
        baseline
            til 100000000
        behaviour
            til 1

Benchmark blocks appear at the same level as should blocks. The simple case above states that the code til 1 should be at least as fast as til 100000000.

Setup and teardown of a benchmark is allowed as well, as in:

feature
   bench
       setup
           .ab.num : 1
       baseline
           til 100000000
       behaviour
           til .ab.num
       teardown
           delete num from `.ab

For more information about benchmark blocks, refer to qu in the Function Reference available from the Help menu.

Running tests outside of Developer

qCumber tests can be run from the command line using the qcumber script.

qcumber.png

See the qCumber section in q build utilities documentation for more information.