Q build utilities
Some libraries are made available as scripts to aid in using these outside the Analyst UI. These scripts make the use of these libraries much easier within automated build systems.
The libraries currently available as scripts include:
- qdoc - q documentation generator
- qlint - q static code linter
- qcumber - q unit and property test framework
- axrepo - convert Analyst
kxscm/repositories to q script files
Instead of starting a q process and using
\l to load the library, any of the above libraries
which are loaded directly when starting q will treat the library as a command-line script. See
the libraries overview for the associated script files and necessary environment
Running any library script with the
-help flag will output help text. For example:
q $ANALYST_HOME/ws/qcumber.q_ -help
The output of the
-help flag for each script is included below for convenience. Optional
flags are enclosed in
qdoc - documentation generation
$ q $ANALYST_HOME/ws/qdoc.q_ -help qDoc: q documentation generator Usage: q qdoc.q_ <flags> Flags: -help - display this help -src <dir/file> - path to q scripts to document (allows multiple -src flags) -out <dir> - path to store generated documentation [-render] - configure markdown files to be used with mkdocs renderer [-map <from>:<to>] - use a non-default tag name, i.e., -map description:fileoverview [-exclude] <regex> - items with a name matching the regex will not be documented [-index] <file> - use this as the index file when rendering markdown using mkdocs [-group] <option> - (default category) use this to change item groupings in markdown files - supported options: category, subcategory [-docprivate] - (default off) include items with an @private tag [-docempty] - (default off) include items that have no qdoc tags [-doctodo] - (default off) include items that have @todo tags [-norecurse] - (default off) do not recurse into subdirectories looking for q scripts [-version] - print the library version
qlint - q static linter
$ q $ANALYST_HOME/ws/qlint.q_ -help qLint: q source code linter Usage: q qlint.q_ <flags> Flags: -help - display this help -src <dir/file> - path to q script(s) [-out] <dir> - path to a file to store results - files supported: .json, .csv, .dat (q serialized) [-level] <level> - (default info) lint level - one of error, warning, info [-version] - print the library version
qcumber - test framework
$ q $ANALYST_HOME/ws/qcumber.q_ -help qCumber: q test runner Usage: q qcumber.q_ <flags> Flags: -help - display this help -src <dir> - path to a q script to load before running tests -test <dir/file> - path to a quke file or directory or quke files to test [-out] <dir> - path to a file to store results - files supported: .json, .dat (q serialized), .xml (junit) [-times] <number> - number of times to run each property block [-color] - colorize the output [-version] - print the library version
Example: using in automated builds
This example demonstrates the use of the above script utilities in an example build of a repository
version-controlled in Analyst
kxscm/ format. The example here would work equivalently with
a repository version-controlled as regular q scripts, simply by skipping the
step (step 3 below).
The following outlines a build-and-test workflow of the above repositories outside of the Analyst UI.
Step 1: set up environment variables
As outlined in the libraries overview, the environment variables pointing to Analyst must be set.
Step 2: clone the repositories
$ git clone <origin>/project-src
Step 3: convert to q scripts
If using modules in Analyst, the artifacts are saved in a custom
kxscm format which
stores each function and data item separately. When releasing the project, this format needs to be
converted to q scripts rather than the custom
axrepo.q_ will perform this conversion.
This step would not be required if the repositories were already in script format (i.e., not using Analyst modules).
$ q $ANALYST_HOME/ws/axrepo.q_ -src ./project-src -out ./workspace/out/project-src
The above commands create a
workspace/out/ directory with the contents of the repositories as regular q scripts and files.
Step 4: lint the sources
Linting helps identify errors in q scripts statically. In many cases,
error level issues identify
problems in the script that would cause some form of error at runtime. We can run the linter and generate a
report by running the below:
$ q $ANALYST_HOME/ws/qlint.q_ \ -src ./workspace/out/project-src/ \ -out ./workspace/reports/lint.dat
The above creates a
workspace/reports/ directory, and saves the result of linting all q files and qcumber tests in the
out directory in a q serialized file
lint.dat. We save to a file to aid in processing a final report after
all steps of this automation have been completed.
Alternatively, if running interactively, it’s also possible to retrieve a readable output if the
out flag is not
specified, shown below.
$ q $ANALYST_HOME/ws/qlint.q_ -src ./workspace/out/project-src Starting qLint: q source code linter Linting q sources: workspace/out/ warning workspace/out/project-src/.math.geo.q:27 - unreachable_code ": c" info workspace/out/project-test/.math.geo.test/benchmark.quke:3 - redundant_global_assign ".math.cities" workspace/out/project-test/.math.geo.test/benchmark.quke:8 - redundant_global_assign ".math.table" workspace/out/project-test/.math.geo.test/haversine.quke:3 - redundant_global_assign ".math.cities"
Step 5: run the tests
All qcumber tests can be run by pointing
qcumber.q_ to the top-level q script and test
directory. The script pointed to by the
-src flag will be loaded before running any tests.
$ q $ANALYST_HOME/ws/qcumber.q_ \ -src ./workspace/out/project-src/load.q \ -test ./workspace/out/project-src/ \ -out ./workspace/reports/test.dat
A report will be created next to the lint report as a q serialized file. Again, if running interactively,
it’s also possible to retrieve a readable output if the
out flag is not specified.
$ q $ANALYST_HOME/ws/qcumber.q_ \ -src ./workspace/out/project-src/load.q \ -test workspace/out/project-src/
Starting qCumber: q test runner Loading src: workspace/out/project-src/.math/load.q Running tests: workspace/out/project-test/ benchmark - pass | (bench) basic query performance haversine work for a variety of co-ordinate pairs - pass | (should) a specific result handle extreme coordinates - pass | (should) it to "wrap around" the international date line - pass | (should) the distance between poles to be half the Earth's circumference all results should be less than half the circumference of the Earth - pass | (property) atan2 calculate atan2 - pass | (should) a specific result degToRad calculate radians from degrees - pass | (should) 90 degrees to be correct - pass | (should) 180 degrees to be correct - pass | (should) 270 degrees to be correct - pass | (should) 360 degrees to be correct - pass | (should) 0 degrees to be correct square square a number - pass | (should) a specific result
Step 6: generate documentation
Finally, once the linter and tests are passing, accompanying documentation can be generated if using
qdoc.q_ utility can generate Markdown or
q $ANALYST_HOME/ws/qdoc.q_ \ -src ./workspace/out/project-src/ \ -out ./workspace/qdoc -render
-render flag was given, the output in
./workspace/qdoc/doc will be an MkDocs-ready directory. With
mkdocs installed, running
mkdocs serve would generate the following output.
If using MkDocs output, it is up to the user to use MkDocs to build the final HTML documentation, or package in any other way desired.
Following this example, the following directory structure has been created.
outdirectory contains the converted
kxscmrepositories as q scripts
qdocdirectory contains both Markdown and an
reportsdirectory contains the results of the
qcumbertests and the
$ tree -a workspace workspace/ ├── out │ └── project-src │ ├── .math │ │ └── load.q │ ├── .math.geo.q │ ├── .math.geo.test │ │ ├── benchmark.quke │ │ └── haversine.quke │ ├── .math.geo.test.q │ ├── .math.q │ ├── .math.trig.q │ ├── .math.trig.test │ │ ├── atan2.quke │ │ ├── degToRad.quke │ │ └── square.quke │ └── .math.trig.test.q ├── qdoc │ ├── doc │ │ ├── docs │ │ │ ├── index.md │ │ │ └── math.md │ │ └── mkdocs.yml │ └── md │ └── math.md └── reports ├── lint.dat └── test.dat