qpk
this document describes everything1 you might want to know about qpk files.
a qpk is a binary redistributable of a q-packer built component.
it can be used to implement a base system, or with some care it can be used as a dependency in another q-packer built project. with some effort they can be also loaded dynamically.
the qpk file format
informally, the qpk file is a zip file. somewhere2 inside that zipfile is a qpk.version.txt
and it is the directory that is in (including the root of the zipfile) that contains a file
called startq.q
there may be multiple qpk.version.txt
files, but they must all share a common parent directory
which contains a qpk.version.txt
file, thus this is valid:
a/qpk.version.txt
a/b/qpk.version.txt
a/b/startq.q
a/startq.q
but this is not:
a/qpk.version.txt
a/startq.q
b/qpk.version.txt
b/startq.q
the startq.q
script must be loaded from that directory and never with an absolute path; that is
you need to do something like this:
{c:system"cd";system"cd /path/to/a";system"l start.q";system"cd ",c;}[]
and not:
system"l /path/to/a/start.q";
qpk customisable base
one use case of qpk is as a "base image" for a system that is designed for customisation.
this is useful for an application that is designed to be customised with a little bit of extra q-code.
having a "customisable base" in a library component is not recommended since this effectively changes the language, and may make it very difficult to integrate your component with another q-based system.
qpk as dependencies
q doesn't have any modular encapsulation, so you must choose your names wisely.
KX recommends underscore-separated reverse DNS names, e.g. .com_kx_Project
for components intended
to be consumed by others.
q also doesn't permit resource sharing: .z.ts
is not a shared resource to be covered, but
the callback for the application. if you need a regular event you could ask the consumer of your
component to call a function periodically, or you could create a subprocess with \q
that has its
own timer and communicates back to this process via IPC.
q applications tend to be divided into three categories:
- real-time processes
- feed-time processes
- user-time processes
the first one has hard real-time requirements, and so your module cannot be used in a real-time
process if it ever blocks. if you have need for blocking/synchronisity, offload those concerns
to a subprocess started with \q
3 just as you would for .z.ts
the second type is typically driven by a reliable feed (such as a ticker plant) and so modules inside
must be entirely deterministic. this means you should not use any operations that may modify determinism
such as rand
or ?
unless it is driven entirely by input. processes can receive input from multiple
partners, but one (and only one!) should be the feed. again, if you need \S
or some kind of non-
deterministic events to occur, move them to a subprocess and provide an interface for the consumer to
tell you how to add new events to the feed (i.e. where is the ticketplant).
the third type are typically "query processes" that are leased by users temporarily as a "seat" to
run queries from. the consequence of this is you cannot expect a process to be able to hopen you,
or even that networking will be available. if you need q-ipc, offload that concern to a subprocess
started with \q
.
the astute reader will note much of the advice centers around this \q
pattern, so it should be
no surprise that most qpk entry points are nothing more than: 2:
bindings for C extensions,
definition of helper functions, and the startup of subprocesses. some useful patterns are documented.
writing module helpers
subprocess helpers are started with \q
(system"q"). if they are fully async, they can be started
at any time. if they aren't, starting the q-based helper should be done at the top-level of the
entrypoint and at no other time.
hdel`:/tmp/startup_helper;
system"q helper.q -p 0W -reg /tmp/startup_helper";
while[@[{ child::hopen get `:/tmp/startup_helper; 0b}; []; 1b]];
at this point, you can communicate with child. if access to child is critical, consider chaining .z.pc to detect failure:
.z.pc:{if[x~z;'"helper.q exited"];y x}[;.z.pc;child];
the helper.q
that you might write, should start by recording the registration:
set[hsym`$first .Q.opt[.z.x]`reg]`$":unix://",string system"p";
qpk loading
a qpk file can be loaded dynamically. the qpk
tool included with q-packer can unzip and mount
the qpk file in a predictable way.
\mkdir /tmp/mountpoint
\cd /tmp/mountpoint
`:qp.json 0: enlist .j.j (1#`default)!enlist (1#`depends)!enlist enlist"f"
`:f.qpk 1: {bytes for qpk here...}
\qpk -xd
\cd qproot/default/f
\l startq.q
if you want to load the qpk file without using the qpk
tool, the sequence is slightly different:
\mkdir /tmp/mountpoint
\cd /tmp/mountpoint
`:f.qpk 1: {bytes for qpk here...}
\unzip f.qpk
system "cd ",first "/qpk.version.txt" vs raze system"find . -name qpk.version.txt|sort|tail -n1";
\l startq.q
NB) we would like to support other processes in the future, but these two will always work.
-
hopefully. ↩
-
Since q-packer 1.1.0 (XXX confirm) this is the root directory, however qpk files built with older versions may find the files prefixed with some other directory names. The location of the
qpk.version.txt
marks the correct junction point wherestartq.q
will be found. ↩ -
an exception is for module loading. if you need to wait for something to occur before startup, for example for one of the subprocesses to be ready, do it at load-time (the toplevel of the main entry point) instead of in response to a library call. ↩