Skip to content

Using QPacker

A packaging utility for deploying kdb+ applications to the cloud

Dependencies

Core:

Linux
Docker
jq (Linux JSON parser) Version 1.6+
zip/unzip
cpio
OpenSSL

Optional:

base64
Docker Compose

Requirements

Docker should be able to run under your user account without using sudo:

Your host should have access to the internet to pull Docker images, or access should be configured to allow Docker through a proxy.

Docker must run on the target, not where the target is a container with Docker running on the host

Installation

Before starting development, install QPacker in your environment using the self-extracting installation script.

Usage:

./installer [dir] [option]

[dir]:                      Directory to install to. No options may be given.
                            Will prompt for directory if not specified
                            Will default to /usr/local for root
                            Will default to $HOME/.local for user
                            Directory must have bin, and lib directories below it
                            Binary will be placed in [dir]/bin
                            [dir]/bin must be in PATH

options:
   [--uninstall]:           Uninstall qpacker
   [--receipt]:             List all files currently installed
   [--replace]:             Force install of version on top of current  
   [--help | --usage]:      Print usage

When uninstalling or replacing, the installer will check PATH for qp, along with default install directories. If not found a normal install will proceed.

Licenses

QPacker will generate an on-demand license the first time it runs a build step that requires one. For this, configure your global Git user.name and user.email as follows:

git config --global user.name "FIRST_NAME LAST_NAME"
git config --global user.email "MY_NAME@example.com"

QPacker passes these details to the license-generation script and you receive an email with an activation link: click it.

Copy the license provided as part of the EAP to your ~/.qp.licenses folder. QPacker will look there to determine which license should be used.

Alternatively if you already have a license you wish to use it can be passed to QPacker via an environment variable containing a base64 encoding of the license file, e.g.

export KDB_LICENSE_B64=$(base64 -w0 kc.lic)

or for k4.lic style licenses

export KDB_K4LICENSE_B64=$(base64 -w0 k4.lic)

Images

By default QPacker will build a Docker image for your application. This image can then be tagged and pushed to a Docker registry of your choice. The images created are set to run as non-root user (USER nobody), this means that you cannot modify the contents of the container when it is running. If you have code running inside the container which needs to write data, then the location being written to needs to be volume-mapped into the container at run time.

qp run

Once QPacker has sucessfully built a Docker image you can run the application using qp run. This will launch the new container using Docker. The configuration for the container will be passed in via the qpbuild/.env file, which contains a base64-encoded license for kdb+ and the SHA256 hash of the Docker container.

Running the containerized application in this way is useful to debug simple standalone applications. To run multiple images together, consider using Docker Compose or a more fully featured container orchestrator such as Kubernetes.

Docker Compose

To run several images which make up an application, consider using Docker Compose. The basic-tick-system example provides a sample docker-compose.yml which shows how volumes can be mapped into the containers and ports exposed to the base system.

To run any image you need to provide a license file as a base64-encoded string e.g.

environment:
      - KDB_LICENSE_B64="base64 encoded kc.lic string"
      - KDB_K4LICENSE_B64="base64 encoded k4.lic string"

Usage

This usage is also included in the README folder.

qp command and options:

Options Description
build Build a project and all dependencies [default if no args provided].
[target] Specific target to build (e.g. "qp build tp") if omitted, will default to all.
[-gcp] Use HashiCorp packer.io to build an image in GCP.
[-aws] Use HashiCorp packer.io to build an image in AWS.
[-azure] Use HashiCorp packer.io to build an image in Azure.
run Runs an entrypoint for the project, rebuilding if necessary.
[entrypoint] Specific entrypoint to run (e.g. "qp run tp") if omitted, will default to 'default'.
[-binlinux] Run under local kdb+ installation (e.g. "qp run -binlinux tp").
[-no-rebuild] Do not rebuild even if source has changed.
[ -- ] Pass args through to entrypoint (e.g. "qp run tp -- -p 1234").
doctor Analyses the project environment and provides recommendations.
clean Clean up any files created by qp that are in the tree.
tag Tag a docker image of a built target. Target must exist in qpbuild/.env (e.g. "qp tag gcr.io/cloudpak/tp 1.2.3") To tag all images omit the image name from the url and pass the -all argument (e.g. "qp tag gcr.io/cloudpak 1.2.3 -all")
push Push a tagged docker image or a qpk file. image: (e.g. "qp push gcr.io/cloudpak/tp 1.2.3") qpk: (e.g. "qp push gitlab.com/example/my-app/myapp.qpk 0.0.1") NB qpk will be picked up from "./qpbuild/qpk/"
pull Pull a pre-built qpk to be used as a dependency in your project.(e.g. "qp pull gitlab.com/example/my-app/myapp.qpk 0.0.1") NB qpk will be placed in current directory
$HOME/.qp.licenses/

Contains KX license information as prompted for on your first run of QPacker. If a kc.lic or k4.lic license file is here it will be used for all build or run targets.

.qpignore

Patterns (in much the same format as .gitignore might have) that can be used to omit source code and other artifacts from the qpbuild tree. Can be placed in any project root directory as a sibling of qp.json.

QPDEBUG

To produce debug logging (set -x), set environment variable QPDEBUG to 1.

You can then capture output to file and to screen via qp build 2>&1 | tee -a qp.out.

To turn off debug logging either unset QPDEBUG or export QPDEBUG=0. NB: with debug enabled, QPacker will not clean up files in /tmp created by the build process.

QPK artifact

Building

The qp build command creates a QPK artifact as an output. It is a ZIP file and can be distributed. QPacker can use it in lieu of a dependency in the qp.json file. The QPK can be found in the qpbuild/qpk folder of your repository after building. To reset the folder in preparation for a clean build, use qp clean.

The QPK contains all of the files, but any dependency can also have a .qpignore file with patterns (much like .gitignore might have) used to omit source code and other artifacts from the build tree.

qp.json

The root directory of every QP project contains a qp.json file, with metadata for the package: what is it called, dependencies and the location for entrypoints for the qp run command.

The base level of the qp.json will be each component name, under which are properties. Properties QPacker parses are:

property value
entry location of the entrypoint for the qp run command (string array)
depends dependencies of the application (string array)
docker_base_image base image to build the Docker container, default: QPDOCKER_BUILD_BASE

For example, a project has a dependency on kurl.qpk. QPacker searches for the dependency in

  1. folders relative to the current project directory; failing that
  2. relative to the directories listed in the QPPATH environment variable; failing that
  3. relative to the installation directory of QPacker itself.

For example, a project has a dependency on bq.qpk. QP searches for the dependency in folders relative to the current project directory; failing that, relative to the directories listed in the QPPATH environment variable; failing that, relative to the installation directory of QP itself. The qp.json file for this application would be

{
  "default": {
    "depends": [ "bq" ],
    "entry": [ "myapp.q" ]
  }
}

A qp.json can contain multiple applications, for example

{
  "rdb":{
    "entry": [ "rdb.q"]
  },
  "tp":{
    "entry": [ "tp.q" ]
  }
}

QPPATH separators vary by operating system

QPPATH items are separated by colon : on Linux and macOS, but by semicolon ; on Windows.

It is designed for use by CI/CD tools, and not for ordinary development.

Publishing

Substitute gcr.io/myproject for Google Cloud, or dkr.ecr.{region}.amazonaws.com for Amazon, or whatever registry you use.

Software is often built by disparate teams, who may individually have remit over a particular component, and package that component for consumption by others. QPacker will store all artifacts for a project in a QPK file. While this file is intended for binary dependencies, it is also intended to be portable across environments. The QPK artifact can be published to the package registry on Git using

qp push gitlab.com/user/my-app/bq.qpk 0.1.0

This could also be added to your CI pipeline as a post-commit hook.

Downloading

Once the QPK artifact has been published to a registry, other projects can pull it for inclusion in their applications. Starting a new project based on QPacker and using the REST library can be as simple as:

mkdir my-app
cd my-app
echo '{ "default": { "entry": [ "app.q" ], } }' > qp.json
qp pull gitlab.com/user/my-app/bq.qpk 0.1.0

Docker

Container registry

By default, QPacker generates a Docker image for each application it builds. The images can be pushed to Docker Hub-compatible container repositories, including private repositories such as Google Cloud or GitLab. You can use:

qp tag registry.gitlab.com/user/my-app/bq 0.1.0
qp push registry.gitlab.com/user/my-app/bq 0.1.0

Docker directory permissions

By default qp build runs Docker images as the root user and generates output (often to ./qpbuild) which will be owned by root but the permissions will be altered via chmod -R 777.

For all QPacker-related output to be generated as your own user, configure Docker with userns-remap:

  1. Get your user id $ id -u e.g. 1000
  2. Get your users gid $ id -g e.g. 1000
  3. Edit /etc/subuid and add username:1000:65536
  4. Edit /etc/subgid and add username:1000:65536
  5. Create or edit /etc/docker/daemon.json
  6. Add (optional for Docker-on-Windows and Docker-on-Mac)

    {
        "userns-remap": "username"
    }

  7. Restart the Docker daemon using systemctl or similar

This change will make your current Docker containers inaccessible

Many Docker/Kubernetes implementations will prefer the default Alpine-based image for its small size and fast booting time, but compliance may require another image, so the “base” image can be overridden by setting the QPDOCKER_BASE environment variable to the Docker image to use in the FROM Docker stanza.

If you need to specify this for some applications and not others, you can specify the docker_base_image key alongside entry and depends.

The QPDOCKER_BASE environment variable controls the Linux distro QPacker uses for build and run. By default this is set to centos:8. This can be changed before running qp by setting `export QPDOCKER_BASE=ubuntu:20.04. The distros supported are Centos and Ubuntu.

To build a specific dependency under a particular distro, set the following in the qp.json inside the dependency source tree

"docker_base_image": [ "centos:8" ]

Cloud VM Builder

QPacker can interface with Hashicorp Packer to generate virtual machine (VM) images for AWS, GCP and Azure. These VM images can then be used as templates for a VM instance running in the cloud.

When a cloud target (-aws,-gcp,-azure) is passed to QPacker, an image is generated for each application defined in the top-level qp.json file. The QPK file resulting from each application is installed into the image and integrated with systemd to allow the startq.sh launch script to start the application on boot.

When qp build is first run with a cloud target (-aws,-gcp,-azure) it copies the default packer config into $HOME/.qp.config/packer/. QPacker exits at this point to let you edit the config to include your own account information. Once in place, QPacker can be rerun and it will try to complete the VM image builds from the qp.json.

The default configuration

$HOME/.qp.config/packer/qp_packer.json

contains builders for the three main cloud vendors but you can add further builders: update

$HOME/.qp.config/packer/qp_packer_vars.json

with information about your cloud provider account.

By default QPacker looks for cloud-vendor credentials in

AWS      $HOME/.aws/credentials                                 
GCP      $HOME/.config/gcloud/application_default_credentials.json 
Azure    $HOME/.azure/azureProfile.json 

See your cloud provider’s documentation for how to configure the credentials.

Cloud VM licenses

To use cloud provider VM images built with QPacker, provide a license file to each instance of the image. The current approach encodes the license file base64 and passes it into the VM instance as instance metadata. The application start script startq.sh reconstructs the file and places it in the QLIC location.

Configuration for the instance meta data:

       metadata field/key       metadata value
------------------------------------------------------------------
AWS    User data                $(base64 -w0 $HOME/.qp.licenses/kc.lic)
GCP    kx-ondemand-metadata     $(base64 -w0 $HOME/.qp.licenses/kc.lic)
Azure  Custom data              $(base64 -w0 $HOME/.qp.licenses/kc.lic)

Using QPacker in air-gapped environments

To use QPacker in air-gapped environments where there is no access to the internet, prebuild the three builder Docker images qlocker, sobuilder, pybuilder.

The environment used to prepare these images will require access to the internet to pull down their dependencies

Once built, the images should be made available to QPacker via an internal Docker registry. For QPacker to pick them up automatically, tag them with the version number returned by qp version.

To simplify this process a qpacker-images TGZ is available for download, along with each release of QPacker. It contains all the files required to build the three builder images, and an example runtime image for the resulting QPacker Docker artifacts.

Package contents

The qpacker-images release TGZ expands into the following layout:

qpacker-images-X.Y.Z/
├── build-qpacker-images.sh
├── py/
├── q/
├── run/
└── so/

Build environment

The host used to build the images should be a Linux platorm with Docker installed and access to the following sites:

https://hub.docker.com/ 
http://dl-cdn.alpinelinux.org/

Build preparation

Prepare the build environment by unpacking the qpacker-images TGZ and changing to the resulting directory.

tar -xvzf qpacker-images-X.Y.Z.tgz
cd qpacker-images-X.Y.Z

Custom content

How custom content, such as environment variables, can be added to the beginning of the Dockerfiles used to build the images

Create a file called docker.include.txt inside the qpacker-images-X.Y.Z directory. The contents of this file will be appended to each of the Dockerfiles (directly below the FROM) used to build qlocker, sobuilder, pybuilder and the runtime image.

For example:

$ cat docker.include.txt
ENV http_proxy=http://myproxy.net:8443
ENV http_proxy=http://myproxy.net:8443

This will have the following included in each Dockerfile

$ cat q/Dockerfile.run
FROM alpine:3.14.0
ENV http_proxy=http://myproxy.net:8443
ENV http_proxy=http://myproxy.net:8443

Building

To build the images:

./build-qpacker-images.sh

Artifacts

On successful execution of the build script there should be a new dist directory in the tree, containing Docker images saved (via docker save) as .tar.gz files:

$ ls -1 dist/
kxbase_2021.05.17_2ce20.tar.gz
pybuilder_2021.05.17_2ce20.tar.gz
qlocker_2021.05.17_2ce20.tar.gz
sobuilder_2021.05.17_2ce20.tar.gz 

The images contain the appriate tag (2021.05.17_2ce20) from the version of QPacker the qpacker-images TGZ was shipped with.

Each tarball contains an image, used by QPacker:

image description
kxbase base runtime image: can be used as the base image for the Docker image artifact produced by QPacker
pybuilder Python builder image, used by QPacker to build an embedPy share object for applications with Python dependencies
qlocker general-purpose container for QPacker build stages, to obfuscate q code and assemble contents for QPacker’s QPK artifacts
sobuilder multi-purpose shared-object builder: can build C/C++ shared objects and understands make, cmake, configure etc.

Loading images

Each of the images above can be loaded via docker load and published to an internal Docker registry for use with QPacker.

For QPacker to pick up the image it must be present on the host machine (via docker pull) and it must also be tagged with the version number from the image .tar.gz file.

The images are loaded as follows:

docker load < kxbase_2021.05.17_2ce20.tar.gz
docker load < pybuilder_2021.05.17_2ce20.tar.gz
docker load < qlocker_2021.05.17_2ce20.tar.gz
docker load < sobuilder_2021.05.17_2ce20.tar.gz

This results in the following entries in the local Docker images cache:

$ docker images
REPOSITORY   TAG                IMAGE ID       CREATED          SIZE
kxbase       2021.05.17_2ce20   f81c9b14aa78   16 minutes ago   62.7MB
sobuilder    2021.05.17_2ce20   09a66bbdde41   16 minutes ago   255MB
pybuilder    2021.05.17_2ce20   9d9a41d46dbe   17 minutes ago   271MB
qlocker      2021.05.17_2ce20   718b7c994724   19 minutes ago   222MB

These images can then be retagged and pushed into another Docker registry, and QPacker can use them provided the TAG matches the version returned via qp version.

For example, for a registry registry.mycompany.com/kx/qpacker/ the tag and push commands would be:

docker tag kxbase:2021.05.17_2ce20 \
  registry.mycompany.com/kx/qpacker/kxbase:2021.05.17_2ce20
docker push registry.mycompany.com/kx/qpacker/kxbase:2021.05.17_2ce20

docker tag sobuilder:2021.05.17_2ce20 \
  registry.mycompany.com/kx/qpacker/sobuilder:2021.05.17_2ce20
docker push registry.mycompany.com/kx/qpacker/sobuilder:2021.05.17_2ce20

docker tag pybuilder:2021.05.17_2ce20 \
  registry.mycompany.com/kx/qpacker/pybuilder:2021.05.17_2ce20
docker push registry.mycompany.com/kx/qpacker/pybuilder:2021.05.17_2ce20

docker tag qlocker:2021.05.17_2ce20 \
  registry.mycompany.com/kx/qpacker/qlocker:2021.05.17_2ce20
docker push registry.mycompany.com/kx/qpacker/qlocker:2021.05.17_2ce20

This image is then picked up by QPacker on any host that has pulled the image into its image cache:

$ docker pull registry.mycompany.com/kx/qpacker/kxbase:2021.05.17_2ce20
$ docker pull registry.mycompany.com/kx/qpacker/sobuilder:2021.05.17_2ce20
$ docker pull registry.mycompany.com/kx/qpacker/pybuilder:2021.05.17_2ce20
$ docker pull registry.mycompany.com/kx/qpacker/qlocker:2021.05.17_2ce20
$ qp build
... [output trimmed]
INFO  | Build   | Using pre-built image qlocker:2021.05.17_2ce20
INFO  | Build   | Using pre-built image sobuilder:2021.05.17_2ce20
INFO  | Build   | Using pre-built image pybuilder:2021.05.17_2ce20

To use the pre-built kxbase image, set:

export QPDOCKER_BASE="registry.mycompany.com/kx/qpacker/kxbase:2021.05.17_2ce20"

The resulting image should be built on top of the kxbase image.