Skip to content

qpk

this document describes everything1 you might want to know about qpk files.

a qpk is a binary redistributable of a q-packer built component.

it can be used to implement a base system, or with some care it can be used as a dependency in another q-packer built project. with some effort they can be also loaded dynamically.

the qpk file format

informally, the qpk file is a zip file. somewhere2 inside that zipfile is a qpk.version.txt and it is the directory that is in (including the root of the zipfile) that contains a file called startq.q

there may be multiple qpk.version.txt files, but they must all share a common parent directory which contains a qpk.version.txt file, thus this is valid:

a/qpk.version.txt
a/b/qpk.version.txt
a/b/startq.q
a/startq.q

but this is not:

a/qpk.version.txt
a/startq.q
b/qpk.version.txt
b/startq.q

the startq.q script must be loaded from that directory and never with an absolute path; that is you need to do something like this:

{c:system"cd";system"cd /path/to/a";system"l start.q";system"cd ",c;}[]

and not:

system"l /path/to/a/start.q";

qpk customisable base

one use case of qpk is as a "base image" for a system that is designed for customisation.

this is useful for an application that is designed to be customised with a little bit of extra q-code.

having a "customisable base" in a library component is not recommended since this effectively changes the language, and may make it very difficult to integrate your component with another q-based system.

qpk as dependencies

q doesn't have any modular encapsulation, so you must choose your names wisely. KX recommends underscore-separated reverse DNS names, e.g. .com_kx_Project for components intended to be consumed by others.

q also doesn't permit resource sharing: .z.ts is not a shared resource to be covered, but the callback for the application. if you need a regular event you could ask the consumer of your component to call a function periodically, or you could create a subprocess with \q that has its own timer and communicates back to this process via IPC.

q applications tend to be divided into three categories:

  • real-time processes
  • feed-time processes
  • user-time processes

the first one has hard real-time requirements, and so your module cannot be used in a real-time process if it ever blocks. if you have need for blocking/synchronisity, offload those concerns to a subprocess started with \q3 just as you would for .z.ts

the second type is typically driven by a reliable feed (such as a ticker plant) and so modules inside must be entirely deterministic. this means you should not use any operations that may modify determinism such as rand or ? unless it is driven entirely by input. processes can receive input from multiple partners, but one (and only one!) should be the feed. again, if you need \S or some kind of non- deterministic events to occur, move them to a subprocess and provide an interface for the consumer to tell you how to add new events to the feed (i.e. where is the ticketplant).

the third type are typically "query processes" that are leased by users temporarily as a "seat" to run queries from. the consequence of this is you cannot expect a process to be able to hopen you, or even that networking will be available. if you need q-ipc, offload that concern to a subprocess started with \q.

the astute reader will note much of the advice centers around this \q pattern, so it should be no surprise that most qpk entry points are nothing more than: 2: bindings for C extensions, definition of helper functions, and the startup of subprocesses. some useful patterns are documented.

writing module helpers

subprocess helpers are started with \q (system"q"). if they are fully async, they can be started at any time. if they aren't, starting the q-based helper should be done at the top-level of the entrypoint and at no other time.

hdel`:/tmp/startup_helper;
system"q helper.q -p 0W -reg /tmp/startup_helper";
while[@[{ child::hopen get `:/tmp/startup_helper; 0b}; []; 1b]];

at this point, you can communicate with child. if access to child is critical, consider chaining .z.pc to detect failure:

.z.pc:{if[x~z;'"helper.q exited"];y x}[;.z.pc;child];

the helper.q that you might write, should start by recording the registration:

set[hsym`$first .Q.opt[.z.x]`reg]`$":unix://",string system"p";

qpk loading

a qpk file can be loaded dynamically. the qpk tool included with q-packer can unzip and mount the qpk file in a predictable way.

\mkdir /tmp/mountpoint
\cd /tmp/mountpoint
`:qp.json 0: enlist .j.j (1#`default)!enlist (1#`depends)!enlist enlist"f"
`:f.qpk 1: {bytes for qpk here...}
\qpk -xd
\cd qproot/default/f
\l startq.q

if you want to load the qpk file without using the qpk tool, the sequence is slightly different:

\mkdir /tmp/mountpoint
\cd /tmp/mountpoint
`:f.qpk 1: {bytes for qpk here...}
\unzip f.qpk
system "cd ",first "/qpk.version.txt" vs raze system"find . -name qpk.version.txt|sort|tail -n1";
\l startq.q

NB) we would like to support other processes in the future, but these two will always work.


  1. hopefully. 

  2. Since q-packer 1.1.0 (XXX confirm) this is the root directory, however qpk files built with older versions may find the files prefixed with some other directory names. The location of the qpk.version.txt marks the correct junction point where startq.q will be found. 

  3. an exception is for module loading. if you need to wait for something to occur before startup, for example for one of the subprocesses to be ready, do it at load-time (the toplevel of the main entry point) instead of in response to a library call.