Skip to content

Using QPacker

A packaging utility for deploying kdb+ applications to the cloud

Quick start

Dependencies

Core:

Linux
Docker
jq (Linux JSON parser)
zip/unzip
rsync
base64
OpenSSL

Optional:

docker-compose

Requirements

Docker should be able to run under your user account without using sudo:

Your host should have access to the Internet to pull Docker images, or access should be configured to allow Docker through a proxy.

Installation

Before starting development, install QPacker in your environment using the self-extracting installation script.

Usage:

./installer [dir] [option]

[dir]:                      Directory to install to. No options may be given.
                            Will prompt for directory if not specified
                            Will default to /usr/local for root
                            Will default to $HOME/.local for user
                            Directory must have bin, and lib directories below it
                            Binary will be placed in [dir]/bin
                            [dir]/bin must be in PATH

options:
   [--uninstall]:           Uninstall qpacker
   [--receipt]:             List all files currently installed
   [--replace]:             Force install of version on top of current  
   [--help | --usage]:      Print usage

When uninstalling or replacing, the installer will check PATH for qp, along with default install directories. If not found a normal install will proceed.

Licenses

QPacker will generate an on-demand license the first time it runs a build step that requires one. For this, configure your global Git user.name and user.email as follows:

git config --global user.name "FIRST_NAME LAST_NAME"
git config --global user.email "MY_NAME@example.com"

QPacker passes these details to the license-generation script and you receive an email with an activation link: click it.

QPacker will look $HOME/.qp.licenses to determine which license should be used.

Usage

This usage is also included in the README folder.

qp command and options:

options description
build build a project and all dependencies; default if no args provided
[target] specific target to build (e.g. "qp build tp") if omitted, will default to all
[-gcp] use HashiCorp packer.io to build an image in GCP
[-aws] use HashiCorp packer.io to build an image in AWS
[-azure] use HashiCorp packer.io to build an image in Azure
run run an entrypoint for the project, rebuilding if necessary
[entrypoint] specific entrypoint to run, e.g. qp run tp

if omitted, default to default
[-binlinux] run under local kdb+ installation, e.g. qp run -binlinux tp
[-no-rebuild] do not rebuild even if source has changed
[ -- ] pass args through to entrypoint, e.g. qp run tp -- -p 1234
doctor analyze the project environment and make recommendations
clean clean up any files created by qp that are in the tree
tag tag a Docker image of a built target

Target must exist in qpbuild/.env, e.g.
qp tag gcr.io/cloudpak/tp 1.2.3.

To tag all images omit the image name from the URL and pass the -all argument, e.g.
qp tag gcr.io/cloupak 1.2.3 -all
push push a tagged Docker image or a QPK file e.g.

image: qp push gcr.io/cloudpak/tp 1.2.3
qpk: qp push gitlab.com/example/my-app/myapp.qpk 0.0.1

NB: QPK will be picked up from ./qpbuild/qpk/
pull pull a pre-built QPK to be used as a dependency in your project, e.g.
qp pull gitlab.com/example/my-app/myapp.qpk 0.0.1.

NB: QPK will be placed in current directory
$HOME/.qp.licenses/

Contains KX license information as prompted for on your first run of QPacker. If a kc.lic or k4.lic license file is here it will be used for all build or run targets.

.qpignore

Patterns (in much the same format as .gitignore might have) that can be used to omit source code and other artefacts from the qpbuild tree. Can be placed in any project root directory as a sibling of qp.json.

QPDEBUG

To produce debug logging (set -x), set environment variable QPDEBUG to 1.

You can then capture output to file and to screen via qp build 2>&1 | tee -a qp.out.

To turn off debug logging either unset QPDEBUG or export QPDEBUG=0. NB: with debug enabled, QPacker will not clean up files in /tmp created by the build process.

QPK artefact

Building

The qp build command creates a QPK artefact as an output. It is a ZIP file and can be distributed. QPacker can use it in lieu of a dependency in the qp.json file. The QPK can be found in the qpbuild/qpk folder of your repository after building. To reset the folder in preparation for a clean build, use qp clean.

The QPK contains all of the files, but any dependency can also have a .qpignore file with patterns (much like .gitignore might have) used to omit source code and other artefacts from the build tree.

qp.json

The root directory of every QP project contains a qp.json file, with metadata for the package: what is it called, dependencies and the location for entrypoints for the qp run command.

The base level of the qp.json will be each component name, under which are properties. Properties QPacker parses are:

property value
entry location of the entrypoint for the qp run command (string array)
depends dependencies of the application (string array)
docker_base_image base image to build the Docker container, default: QPDOCKER_BUILD_BASE

For example, a project has a dependency on kurl.qpk. QPacker searches for the dependency in

  1. folders relative to the current project directory; failing that
  2. relative to the directories listed in the QPPATH environment variable; failing that
  3. relative to the installation directory of QPacker itself.

The qp.json file for this application would be

{
  "default": {
    "depends": [ "kurl" ],
    "entry": [ "myapp.q" ]
  }
}

A qp.json can contain multiple applications, for example

{
  "rdb":{
    "entry": [ "rdb.q"]
  },
  "tp":{
    "entry": [ "tp.q" ]
  }
}

QPPATH separators vary by operating system

QPPATH items are separated by colon : on Linux and macOS, but by semicolon ; on Windows.

It is designed for use by CI/CD tools, and not for ordinary development.

Publishing

Substitute gcr.io/myproject for Google Cloud, or dkr.ecr.{region}.amazonaws.com for Amazon, or whatever registry you use.

Software is often built by disparate teams, who may individually have remit over a particular component, and package that component for consumption by others. QPacker will store all artefacts for a project in a QPK file. While this file is intended for binary dependencies, it is also intended to be portable across environments. The QPK artefact can be published to the package registry on Git using

qp push gitlab.com/user/my-app/kurl.qpk 0.1.0

This could also be added to your CI pipeline as a post-commit hook.

Downloading

Once the QPK artefact has been published to a registry, other projects can pull it for inclusion in their applications. Starting a new project based on QPacker and using the REST library can be as simple as:

mkdir my-app
cd my-app
echo '{ "default": { "entry": [ "app.q" ], } }' > qp.json
qp pull gitlab.com/user/my-app/kurl.qpk 0.1.0

Docker

Container registry

By default, QPacker generates a Docker image for each application it builds. The images can be pushed to Docker Hub-compatible container repositories, including private repositories such as Google Cloud or GitLab. You can use:

qp tag registry.gitlab.com/user/my-app/kurl 0.1.0
qp push registry.gitlab.com/user/my-app/kurl 0.1.0

Docker directory permissions

By default qp build runs Docker images as the root user and generates output (often to ./qpbuild) which will be owned by root but the permissions will be altered via chmod -R 777.

For all QPacker-related output to be generated as your own user, configure Docker with userns-remap:

  1. Get your user id $ id -u e.g. 1000
  2. Get your users gid $ id -g e.g. 1000
  3. Edit /etc/subuid and add username:1000:65536
  4. Edit /etc/subgid and add username:1000:65536
  5. Create or edit /etc/docker/daemon.json
  6. Add (optional for Docker-on-Windows and Docker-on-Mac)

    {
        "userns-remap": "username"
    }

  7. Restart the Docker daemon using systemctl or similar

This change will make your current Docker containers inaccessible

Many Docker/Kubernetes implementations will prefer the default Alpine-based image for its small size and fast booting time, but compliance may require another image, so the “base” image can be overridden by setting the QPDOCKER_BASE environment variable to the Docker image to use in the FROM Docker stanza.

If you need to specify this for some applications and not others, you can specify the docker_base_image key alongside entry and depends.

The QPDOCKER_BASE environment variable controls the Linux distro QPacker uses for build and run. By default this is set to centos:8. This can be changed before running qp by setting `export QPDOCKER_BASE=ubuntu:20.04. The distros supported are Centos and Ubuntu.

To build a specific dependency under a particular distro, set the following in the qp.json inside the dependency source tree

"docker_base_image": [ "centos:8" ]

Cloud VM Builder

QPacker can interface with Hashicorp Packer to generate virtual machine (VM) images for AWS, GCP and Azure. These VM images can then be used as templates for a VM instance running in the cloud.

When a cloud target (-aws,-gcp,-azure) is passed to QPacker, an image is generated for each application defined in the top-level qp.json file. The QPK file resulting from each application is installed into the image and integrated with systemd to allow the startq.sh launch script to start the application on boot.

When qp build is first run with a cloud target (-aws,-gcp,-azure) it copies the default packer config into $HOME/.qp.config/packer/. QPacker exits at this point to let you edit the config to include your own account information. Once in place, QPacker can be rerun and it will try to complete the VM image builds from the qp.json.

The default configuration

$HOME/.qp.config/packer/qp_packer.json

contains builders for the three main cloud vendors but you can add further builders: update

$HOME/.qp.config/packer/qp_packer_vars.json

with information about your cloud provider account.

By default QPacker looks for cloud-vendor credentials in

AWS      $HOME/.aws/credentials                                 
GCP      $HOME/.config/gcloud/application_default_credentials.json 
Azure    $HOME/.azure/azureProfile.json 

See your cloud provider’s documentation for how to configure the credentials.

Cloud VM licenses

To use cloud provider VM images built with QPacker, provide a license file to each instance of the image. The current approach encodes the license file base64 and passes it into the VM instance as instance metadata. The application start script startq.sh reconstructs the file and places it in the QLIC location.

Configuration for the instance meta data:

       metadata field/key       metadata value
------------------------------------------------------------------
AWS    User data                $(base64 $HOME/.qp.license/kc.lic)
GCP    kx-ondemand-metadata     $(base64 $HOME/.qp.license/kc.lic)
Azure  Custom data              $(base64 $HOME/.qp.license/kc.lic)