Skip to content

Using QPacker

A packaging utility for deploying kdb+ applications to the cloud

Dependencies

Core:

Linux
Docker
jq (Linux JSON parser) Version 1.6+
zip/unzip
cpio
OpenSSL

Optional:

base64
Docker Compose

Requirements

Docker should be able to run unde your user account without using sudo:

Your host should have access to the internet to pull Docker images, or access should be configured to allow Docker through a proxy.

Docker must run on the target, not where the target is a container with Docker running on the host

Installation

Before starting development, install QPacker in your environment using the self-extracting installation script.

Usage:

./installer [dir] [option]

[dir]:                      Directory to install to. No options may be given.
                            Will prompt for directory if not specified
                            Will default to /usr/local for root
                            Will default to $HOME/.local for user
                            Directory must have bin, and lib directories below it
                            Binary will be placed in [dir]/bin
                            [dir]/bin must be in PATH

options:
   [--uninstall]:           Uninstall qpacker
   [--receipt]:             List all files currently installed
   [--replace]:             Force install of version on top of current  
   [--help | --usage]:      Print usage

When uninstalling or replacing, the installer will check PATH for qp, along with default install directories. If not found a normal install will proceed.

Licenses

q-packer requires that a valid kdb+ license file is present in the system.

Copy the license provided as part of the EAP to your ~/.qp.licenses folder. QPacker will look there to determine which license should be used.

Alternatively if you already have a license you wish to use it can be passed to qpacker via an environment variable containing a base64 encoding of the license file, e.g.

kx.lic style license

export KDB_KXLICENSE_B64=$(base64 -w0 kx.lic)

kc.lic style license

export KDB_LICENSE_B64=$(base64 -w0 kc.lic)

k4.lic style license

export KDB_K4LICENSE_B64=$(base64 -w0 k4.lic)

Images

By default QPacker will build a Docker image for your application. This image can then be tagged and pushed to a Docker registry of your choice. The images created are set to run as non-root user (USER nobody), this means that you cannot modify the contents of the container when it is running. If you have code running inside the container which needs to write data, then the location being written to needs to be volume-mapped into the container at run time.

qp run

Once QPacker has sucessfully built a Docker image you can run the application using qp run. This will launch the new container using Docker. The configuration for the container will be passed in via the qpbuild/.env file, which contains a base64-encoded license for kdb+ and the SHA256 hash of the Docker container.

Running the containerized application in this way is useful to debug simple standalone applications. To run multiple images together, consider using Docker Compose or a more fully featured container orchestrator such as Kubernetes.

Docker Compose

To run several images which make up an application, consider using Docker Compose. The basic-tick-system example provides a sample docker-compose.yml which shows how volumes can be mapped into the containers and ports exposed to the base system.

To run any image you need to provide a license file as a base64-encoded string e.g.

environment:
      - KDB_KXLICENSE_B64="base64 encoded kx.lic string"
      - KDB_LICENSE_B64="base64 encoded kc.lic string"
      - KDB_K4LICENSE_B64="base64 encoded k4.lic string"

Usage

This usage is also included in the README folder.

qp command and options:

Options Description
build Build a project and all dependencies [default if no args provided]. qp build can support more than one arg at a time.
[target] Specific target(s) to build (e.g. "qp build tp" or "qp build tp gw") if omitted, will default to all.
[-binlinux] Build in-tree linux and qpk.
[-gcp] Use HashiCorp packer.io to build an image in GCP.
[-aws] Use HashiCorp packer.io to build an image in AWS.
[-azure] Use HashiCorp packer.io to build an image in Azure.
[-docker] Build docker images.
[-linux] Build linux installers e.g. rpm, deb, sh. Will default to -binlinux build if UI field is not set in qp.json.
[-windows] Build windows installers e.g. msi, sh. To build successfully UI field in qp.json must be set.
[-rpm] Build rpm+sh.
[-deb] Build deb+sh.
[-sh] Build sh code only.
run Runs an entrypoint for the project, rebuilding if necessary.
[entrypoint] Specific entrypoint to run (e.g. "qp run tp") if omitted, will default to 'default'.
[-binlinux] Run under local kdb+ installation (e.g. "qp run -binlinux tp").
[-no-rebuild] Do not rebuild even if source has changed.
[ -- ] Pass args through to entrypoint (e.g. "qp run tp -- -p 1234").
doctor Analyses the project environment and provides recommendations.
clean Clean up any files created by qp that are in the tree.
tag Tag a docker image of a built target. Target must exist in qpbuild/.env (e.g. "qp tag gcr.io/cloudpak/tp 1.2.3") To tag all images omit the image name from the url and pass the -all argument (e.g. "qp tag gcr.io/cloudpak 1.2.3 -all")
push Push a tagged docker image or a qpk file. image: (e.g. "qp push gcr.io/cloudpak/tp 1.2.3") qpk: (e.g. "qp push gitlab.com/example/my-app/myapp.qpk 0.0.1") NB qpk will be picked up from "./qpbuild/qpk/"
pull Pull a pre-built qpk to be used as a dependency in your project.(e.g. "qp pull gitlab.com/example/my-app/myapp.qpk 0.0.1") NB qpk will be placed in current directory
$HOME/.qp.licenses/

Contains KX license information as prompted for on your first run of QPacker. If a k[xc4].lic license file is here it will be used for all build or run targets.

.qpignore

Patterns (in much the same format as .gitignore might have) that can be used to omit source code and other artefacts from the qpbuild tree. Can be placed in any project root directory as a sibling of qp.json.

QPDEBUG

To produce debug logging (set -x), set environment variable QPDEBUG to 1.

You can then capture output to file and to screen via qp build 2>&1 | tee -a qp.out.

To turn off debug logging either unset QPDEBUG or export QPDEBUG=0. NB: with debug enabled, QPacker will not clean up files in /tmp created by the build process.

QPTASKSET

To pass a taskset value through to containers which run q, set the environment variable QPTASKSET to taskset value

The format should follow the -c mask which specifies a list of processors, e.g. export QPTASKSET=0-3 would result in q being run with the following taskset stanza tasket -c 0-3.

To turn off QPTASKSET use unset QPTASKSET.

QPK artefact

Building

The qp build command creates a QPK artefact as an output. It is a ZIP file and can be distributed. QPacker can use it in lieu of a dependency in the qp.json file. The QPK can be found in the qpbuild/qpk folder of your repository after building. To reset the folder in preparation for a clean build, use qp clean.

The QPK contains all of the files, but any dependency can also have a .qpignore file with patterns (much like .gitignore might have) used to omit source code and other artefacts from the build tree.

qp.json

The root directory of every QP project contains a qp.json file, with metadata for the package: what is it called, dependencies and the location for entrypoints for the qp run command.

The base level of the qp.json will be each component name, under which are properties. Properties QPacker parses are:

property value
entry location of the entrypoint for the qp run command (string array)
depends dependencies of the application (string array)
docker_base_image base image to build the Docker containers sobuilder/qlocker, default: alpine:3.14.6

For example, a project has a dependency on kurl.qpk. QPacker searches for the dependency in

  1. folders relative to the current project directory; failing that
  2. relative to the directories listed in the QPPATH environment variable; failing that
  3. relative to the installation directory of QPacker itself.

For example, a project has a dependency on bq.qpk. QP searches for the dependency in folders relative to the current project directory; failing that, relative to the directories listed in the QPPATH environment variable; failing that, relative to the installation directory of QP itself. The qp.json file for this application would be

{
  "default": {
    "depends": [ "bq" ],
    "entry": [ "myapp.q" ]
  }
}

A qp.json can contain multiple applications, for example

{
  "rdb":{
    "entry": [ "rdb.q"]
  },
  "tp":{
    "entry": [ "tp.q" ]
  }
}

QPPATH separators vary by operating system

QPPATH items are separated by colon : on Linux and macOS, but by semicolon ; on Windows.

It is designed for use by CI/CD tools, and not for ordinary development.

Publishing

Substitute gcr.io/myproject for Google Cloud, or dkr.ecr.{region}.amazonaws.com for Amazon, or whatever registry you use.

Software is often built by disparate teams, who may individually have remit over a particular component, and package that component for consumption by others. QPacker will store all artefacts for a project in a QPK file. While this file is intended for binary dependencies, it is also intended to be portable across environments. The QPK artefact can be published to the package registry on Git using

qp push gitlab.com/user/my-app/bq.qpk 0.1.0

This could also be added to your CI pipeline as a post-commit hook.

Downloading

Once the QPK artefact has been published to a registry, other projects can pull it for inclusion in their applications. Starting a new project based on QPacker and using the REST library can be as simple as:

mkdir my-app
cd my-app
echo '{ "default": { "entry": [ "app.q" ], } }' > qp.json
qp pull gitlab.com/user/my-app/bq.qpk 0.1.0

Docker

Container registry

By default, QPacker generates a Docker image for each application it builds. The images can be pushed to Docker Hub-compatible container repositories, including private repositories such as Google Cloud or GitLab. You can use:

qp tag registry.gitlab.com/user/my-app/bq 0.1.0
qp push registry.gitlab.com/user/my-app/bq 0.1.0

Docker directory permissions

By default qp build runs Docker images as the root user and generates output (often to ./qpbuild) which will be owned by root but the permissions will be altered via chmod -R 777.

For all QPacker-related output to be generated as your own user, configure Docker with userns-remap:

  1. Get your user id $ id -u e.g. 1000
  2. Get your users gid $ id -g e.g. 1000
  3. Edit /etc/subuid and add username:1000:65536
  4. Edit /etc/subgid and add username:1000:65536
  5. Create or edit /etc/docker/daemon.json
  6. Add (optional for Docker-on-Windows and Docker-on-Mac)

    {
        "userns-remap": "username"
    }

  7. Restart the Docker daemon using systemctl or similar

This change will make your current Docker containers inaccessible

Many Docker/Kubernetes implementations will prefer the default Alpine-based image for its small size and fast booting time, but compliance may require another image, so the “base” image can be overridden by setting the QPDOCKER_BASE environment variable to the Docker image to use in the FROM Docker stanza.

If you need to specify this for some applications and not others, you can specify the docker_base_image key alongside entry and depends.

The QPDOCKER_BASE environment variable controls the Linux distro QPacker uses for image artefacts. By default this is set to alpine:3.14.6. This can be changed before running qp by setting export QPDOCKER_BASE=ubuntu:20.04. The distros supported are Centos and Ubuntu.

The QPDOCKER_BUILD_BASE environment variable controls the Linux distro QPacker uses for the builder containers sobuilder/qpacker/pybiuilder. This controls the Distro for build artefacts and can be changed before running qp by setting `export QPDOCKER_BUILD_BASE=ubuntu:20.04.

To build a specific dependency under a particular distro, set the following in the qp.json inside the dependency source tree

"docker_base_image": [ "rockylinux/rockylinux:8" ]

Using QPacker in air-gapped environments

To use QPacker in air-gapped environments where there is no access to the internet, prebuild the all builder Docker images qlocker, sobuilder, rpmbuilder, debbuilder.

The environment used to prepare these images will require access to the internet to pull down their dependencies

Once built, the images should be made available to QPacker via an internal Docker registry. For QPacker to pick them up automatically, tag them with the version number returned by qp version.

To simplify this process a qpacker-images TGZ is available for download, along with each release of QPacker. It contains all the files required to build the three builder images, and an example runtime image for the resulting QPacker Docker artefacts.

Package contents

The qpacker-images release tgz expands into the following layout:

qpacker-images-X.Y.Z/
├── build-qpacker-images.sh
├── py/
├── q/
├── run/
└── so/

Build environment

The host used to build the images should be a Linux platform with Docker installed and access to the following sites:

https://hub.docker.com/ 
http://dl-cdn.alpinelinux.org/

Build preparation

Prepare the build environment by unpacking the qpacker-images TGZ and changing to the resulting directory.

tar -xvzf qpacker-images-X.Y.Z.tgz
cd qpacker-images-X.Y.Z

Custom content

How custom content, such as environment variables, can be added to the beginning of the Dockerfiles used to build the images

Create a file called docker.include.txt inside the qpacker-images-X.Y.Z directory. The contents of this file will be appended to each of the Dockerfiles (directly below the FROM) used to build qlocker, sobuilder, rpmbuilder, debbuilder and the runtime image.

For example:

$ cat docker.include.txt
ENV http_proxy=http://myproxy.net:8443
ENV http_proxy=http://myproxy.net:8443

This will have the following included in each Dockerfile

$ cat q/Dockerfile.run
FROM alpine:3.14.6
ENV http_proxy=http://myproxy.net:8443
ENV http_proxy=http://myproxy.net:8443

Building

To build the images:

./build-qpacker-images.sh

Artefacts

On successful execution of the build script there should be a new dist directory in the tree, containing Docker images saved (via docker save) as .tar.gz files:

$ ls -1 dist/
kxbase_2021.05.17_2ce20.tar.gz
qlocker_2021.05.17_2ce20.tar.gz
sobuilder_2021.05.17_2ce20.tar.gz 
rpmbuilder_2021.05.17_2ce20.tar.gz 
debbuilder_2021.05.17_2ce20.tar.gz 

The images contain the appriate tag (2021.05.17_2ce20) from the version of QPacker the qpacker-images TGZ was shipped with.

Each tarball contains an image, used by QPacker:

image description
kxbase Base runtime image: can be used as the base image for the Docker image artefact produced by QPacker
qlocker General-purpose container for QPacker build stages, to obfuscate q code and assemble contents for QPacker’s QPK artefacts
sobuilder Multi-purpose shared-object builder: can build C/C++ shared objects and understands make, cmake, configure etc.
rpmbuilder Container for building rpm installers
debbuilder Containers for building debian installers

Loading images

Each of the images above can be loaded via docker load and published to an internal Docker registry for use with QPacker.

For QPacker to pick up the image it must be present on the host machine (via docker pull) and it must also be tagged with the version number from the image .tar.gz file.

The images are loaded as follows:

docker load < kxbase_2021.05.17_2ce20.tar.gz
docker load < qlocker_2021.05.17_2ce20.tar.gz
docker load < sobuilder_2021.05.17_2ce20.tar.gz
docker load < rpmbuilder_2021.05.17_2ce20.tar.gz
docker load < debbuilder_2021.05.17_2ce20.tar.gz

This results in the following entries in the local Docker images cache:

$ docker images
REPOSITORY   TAG                IMAGE ID       CREATED          SIZE
kxbase       2021.05.17_2ce20   f81c9b14aa78   16 minutes ago   62.7MB
sobuilder    2021.05.17_2ce20   09a66bbdde41   16 minutes ago   255MB
qlocker      2021.05.17_2ce20   718b7c994724   19 minutes ago   222MB
rpmbuilder   2021.05.17_2ce20   9aff5fb10b3b   16 minutes ago   526MB
debbuilder   2021.05.17_2ce20   09a66bbdde41   16 minutes ago   139MB

These images can then be retagged and pushed into another Docker registry, and QPacker can use them provided the TAG matches the version returned via qp version.

For example, for a registry registry.mycompany.com/kx/qpacker/ the tag and push commands would be:

docker tag kxbase:2021.05.17_2ce20 \
  registry.mycompany.com/kx/qpacker/kxbase:2021.05.17_2ce20
docker push registry.mycompany.com/kx/qpacker/kxbase:2021.05.17_2ce20

docker tag sobuilder:2021.05.17_2ce20 \
  registry.mycompany.com/kx/qpacker/sobuilder:2021.05.17_2ce20
docker push registry.mycompany.com/kx/qpacker/sobuilder:2021.05.17_2ce20

docker tag qlocker:2021.05.17_2ce20 \
  registry.mycompany.com/kx/qpacker/qlocker:2021.05.17_2ce20
docker push registry.mycompany.com/kx/qpacker/qlocker:2021.05.17_2ce20

docker tag rpmbuilder:2021.05.17_2ce20 \
  registry.mycompany.com/kx/qpacker/rpmbuilder:2021.05.17_2ce20
docker push registry.mycompany.com/kx/qpacker/rpmbuilder:2021.05.17_2ce20

docker tag debbuilder:2021.05.17_2ce20 \
  registry.mycompany.com/kx/qpacker/debbuilder:2021.05.17_2ce20
docker push registry.mycompany.com/kx/qpacker/debbuilder:2021.05.17_2ce20

This image is then picked up by QPacker on any host that has pulled the image into its image cache:

$ docker pull registry.mycompany.com/kx/qpacker/kxbase:2021.05.17_2ce20
$ docker pull registry.mycompany.com/kx/qpacker/sobuilder:2021.05.17_2ce20
$ docker pull registry.mycompany.com/kx/qpacker/qlocker:2021.05.17_2ce20
$ docker pull registry.mycompany.com/kx/qpacker/rpmbuilder:2021.05.17_2ce20
$ docker pull registry.mycompany.com/kx/qpacker/debbuilder:2021.05.17_2ce20
$ qp build
... [output trimmed]
INFO  | Build   | Using pre-built image qlocker:2021.05.17_2ce20
INFO  | Build   | Using pre-built image sobuilder:2021.05.17_2ce20
INFO  | Build   | Using pre-built image rpmbuilder:2021.05.17_2ce20
INFO  | Build   | Using pre-built image debbuilder:2021.05.17_2ce20

To use the pre-built kxbase image, set:

export QPDOCKER_BASE="registry.mycompany.com/kx/qpacker/kxbase:2021.05.17_2ce20"

The resulting image should be built on top of the kxbase image.