Skip to content

Q build utilities

Some libraries are made available as scripts to aid in using these outside the Developer UI. These scripts make the use of these libraries much easier within automated build systems.

The libraries currently available as scripts include:

  • qdoc - q documentation generator
  • qlint - q static code linter
  • qcumber - q unit and property test framework
  • axrepo - convert Developer kxscm/ repositories to q script files

Instead of starting a q process and using \l to load the library, any of the above libraries which are loaded directly when starting q will treat the library as a command-line script. See the libraries overview for the associated script files and necessary environment variables.

Running any library script with the -help flag will output help text. For example:

q $DEVELOPER_HOME/ws/qcumber.q_ -help

The output of the -help flag for each script is included below for convenience. Optional flags are enclosed in [].

qdoc - documentation generation

q $DEVELOPER_HOME/ws/qdoc.q_ -help

qDoc: q documentation generator

Usage: q qdoc.q_ <flags>

Flags:
   -help               - display this help
   -src   <dir/file>   - path to q scripts to document (allows multiple -src flags)
   -out   <dir>        - path to store generated documentation
   [-render]           - configure markdown files to be used with mkdocs renderer
   [-map <from>:<to>]  - use a non-default tag name, i.e., -map description:fileoverview
   [-exclude] <regex>  - items with a name matching the regex will not be documented
   [-index]   <file>   - use this as the index file when rendering markdown using mkdocs
   [-group]   <option> - (default category) use this to change item groupings in markdown files
                           - supported options: category, subcategory
   [-docprivate]       - (default off) include items with an @private tag
   [-docempty]         - (default off) include items that have no qdoc tags
   [-doctodo]          - (default off) include items that have @todo tags
   [-norecurse]        - (default off) do not recurse into subdirectories looking for q scripts
   [-version]          - print the library version

qlint - q static linter

q $DEVELOPER_HOME/ws/qlint.q_ -help

qLint: q source code linter

Usage: q qlint.q_ <flags>

Flags:
    -help               - display this help
    -src     <dir/file> - path to q script(s)
    [-out]   <dir>      - path to a file to store results
                            - files supported: .json, .csv, .dat (q serialized)
    [-level] <level>    - (default info) lint level - one of error, warning, info
    [-version]          - print the library version

qcumber - test framework

q $DEVELOPER_HOME/ws/qcumber.q_ -help

qCumber: q test runner

Usage: q qcumber.q_ <flags>

Flags:
    -help               - display this help
    -src     <dir>      - path to a q script to load before running tests
    -test    <dir/file> - path to a quke file or directory or quke files to test
    [-out]   <dir>      - path to a file to store results
                            - files supported: .json, .dat (q serialized), .xml (junit)
    [-times] <number>   - number of times to run each property block
    [-color]            - colorize the output
    [-version]          - print the library version

Example: using in automated builds

This example demonstrates the use of the above script utilities in an example build of a repository version-controlled in Developer kxscm/ format. The example here would work equivalently with a repository version-controlled as regular q scripts, simply by skipping the axrepo.q_ conversion step (step 3 below).

The following outlines a build-and-test workflow of the above repositories outside of the Developer UI.

Step 1: set up environment variables

As outlined in the libraries overview, the environment variables pointing to Developer must be set.

Step 2: clone the repositories

git clone <origin>/project-src

Step 3: convert to q scripts

If using modules in Developer, the artifacts are saved in a custom kxscm format which stores each function and data item separately. When releasing the project, this format needs to be converted to q scripts rather than the custom kxscm structure. axrepo.q_ will perform this conversion.

NOTE - this step would not be required if the repositories were already in script format (i.e., not using Developer modules).

q $DEVELOPER_HOME/ws/axrepo.q_ -src ./project-src -out ./workspace/out/project-src

The above commands create a workspace/out/ directory with the contents of the repositories as regular q scripts and files.

Step 4: lint the sources

Linting helps identify errors in q scripts statically. In many cases, error level issues identify problems in the script that would cause some form of error at runtime. We can run the linter and generate a report by running the below:

q $DEVELOPER_HOME/ws/qlint.q_ \
    -src ./workspace/out/project-src/ \
    -out ./workspace/reports/lint.dat

The above creates a workspace/reports/ directory, and saves the result of linting all q files and qcumber tests in the out directory in a q serialized file lint.dat. We save to a file to aid in processing a final report after all steps of this automation have been completed.

Alternatively, if running interactively, it's also possible to retrieve a readable output if the out flag is not specified, shown below.

q $DEVELOPER_HOME/ws/qlint.q_ -src ./workspace/out/project-src

Starting qLint: q source code linter

Linting q sources: workspace/out/

warning
    workspace/out/project-src/.math.geo.q:27  -  unreachable_code  ": c"

info
    workspace/out/project-test/.math.geo.test/benchmark.quke:3  -  redundant_global_assign  ".math.cities"
    workspace/out/project-test/.math.geo.test/benchmark.quke:8  -  redundant_global_assign  ".math.table"
    workspace/out/project-test/.math.geo.test/haversine.quke:3  -  redundant_global_assign  ".math.cities"

Step 5: run the tests

All qcumber tests can be run by pointing qcumber.q_ to the top-level q script and test directory. The script pointed to by the -src flag will be loaded before running any tests.

q $DEVELOPER_HOME/ws/qcumber.q_ \
    -src ./workspace/out/project-src/load.q \
    -test ./workspace/out/project-src/ \
    -out ./workspace/reports/test.dat

A report will be created next to the lint report as a q serialized file. Again, if running interactively, it's also possible to retrieve a readable output if the out flag is not specified.

q $DEVELOPER_HOME/ws/qcumber.q_ \
    -src ./workspace/out/project-src/load.q \
    -test workspace/out/project-src/

Starting qCumber: q test runner

Loading src: workspace/out/project-src/.math/load.q
Running tests: workspace/out/project-test/

    benchmark

            - pass | (bench) basic query performance

    haversine
        work for a variety of co-ordinate pairs
            - pass | (should) a specific result

        handle extreme coordinates
            - pass | (should) it to "wrap around" the international date line
            - pass | (should) the distance between poles to be half the Earth's circumference

        all results should be less than half the circumference of the Earth
            - pass | (property)

    atan2
        calculate atan2
            - pass | (should) a specific result

    degToRad
        calculate radians from degrees
            - pass | (should) 90 degrees to be correct
            - pass | (should) 180 degrees to be correct
            - pass | (should) 270 degrees to be correct
            - pass | (should) 360 degrees to be correct
            - pass | (should) 0 degrees to be correct

    square
        square a number
            - pass | (should) a specific result

Step 6: generate documentation

Finally, once the linter and tests are passing, accompanying documentation can be generated if using qdoc annotations. The qdoc.q_ utility can generate Markdown or mkdocs-compatible output.

q $DEVELOPER_HOME/ws/qdoc.q_ \
    -src ./workspace/out/project-src/ \
    -out ./workspace/qdoc -render

Since the -render flag was given, the output in ./workspace/qdoc/doc will be an mkdocs-ready directory. With mkdocs installed, running mkdocs serve would generate the following output.

If using mkdocs output, it is up to the user to use mkdocs to build the final HTML documentation, or package in any other way desired.

Result

Following this example, the following directory structure has been created.

  • the out directory contains the converted kxscm repositories as q scripts
  • the qdoc directory contains both Markdown and an mkdocs-ready site
  • the reports directory contains the results of the qcumber tests and the qlint static linting
tree -a workspace
workspace/
├── out
│   └── project-src
│       ├── .math
│       │   └── load.q
│       ├── .math.geo.q
│       ├── .math.geo.test
│       │   ├── benchmark.quke
│       │   └── haversine.quke
│       ├── .math.geo.test.q
│       ├── .math.q
│       ├── .math.trig.q
│       ├── .math.trig.test
│       │   ├── atan2.quke
│       │   ├── degToRad.quke
│       │   └── square.quke
│       └── .math.trig.test.q
├── qdoc
│   ├── doc
│   │   ├── docs
│   │   │   ├── index.md
│   │   │   └── math.md
│   │   └── mkdocs.yml
│   └── md
│       └── math.md
└── reports
    ├── lint.dat
    └── test.dat