Skip to content

Introduction

What is a kdb Insights Package ?

A package is simply a directory structure consisting of:

  • Configuration to define Deployable kdb Insights Components (e.g. databases, pipelines)
  • Code to define User Defined Analytics to run on those components
  • A single Manifest file to keep track of components, code and dependencies

The packaging toolkit facilitates:

  • Creating and executing custom analytics
  • Deploying entities leveraging those custom analytics

Installation of the packaging tool

KX has produced a package management tool as part of the kdb Insights CLI which should be used to manage packages from development to deployment. Where possible the tool should be used to make changes to your package as it will ensure everything is validated and correctly formatted.

Python, pip and virtual environments

The CLI requires python (>=3.8, <=3.10).

Once installed we recommend using a virtualenv to ensure there are no dependency clashes.

python3 -m venv my-new-pkg-env
source my-new-pkg-env/bin/activate
pip install --extra-index-url https://nexus.dl.kx.com/repository/kxi-pypi-public/simple/ kxicli

For more information on installing the kdb Insights CLI see here.

Configuring the tool

Checking the installation

This section explains how to configure the tool locally for manipulation, installation and storage of package artifacts.

To check that the packaging tool is installed, run:

kxi package --version
{
    "version": "1.8.3",
    "env": {
        "KX_PACKAGE_PATH": null,
        "KX_ARTIFACT_PATH": null,
        "KXI_AUTH_ENABLED": null,
        "INSIGHTS_URL": null,
        "INSIGHTS_CLIENT_SECRET": null,
        "INSIGHTS_CLIENT_ID": null,
        "INSIGHTS_REALM": null
    }
}

Configuration and environment variable precedence

Below are some remarks around specifying values in the shell vs in the config files:

  • Each var can be explicitly set (export KX_PACKAGE_PATH=/some/where)
  • Each var can be explicitly set in its respective config file (see below table)
  • Variables set in shell will take precedence over ones set in config

Configuration and environment variable explained

A tale of two configs

There are two configs that the packaging tool will read:

  • ~/.insights/cli-config
  • ~/.insights/pakx-config
environment variable config key in config description
KX_PACKAGE_PATH pakx-config KX_PACKAGE_PATH Where "Installed" packages will be stored.
KX_ARTIFACT_PATH pakx-config KX_ARTIFACT_PATH Where package archives or "Artifacts" will be stored
KX_DEPLOYMENT_PATH pakx-config KX_DEPLOYMENT_PATH Where "Deployment" objects will be stored
KX_PACKAGE_FORMAT pakx-config KX_PACKAGE_FORMAT The default package format (.yaml by default, can also be .json)
INSIGHTS_URL cli-config hostname The URL to the kdb Insights Enterprise instance to which you want to connect
INSIGHTS_REALM cli-config realm The Insights realm name (defaults to kdb Insights Enterprise)
INSIGHTS_CLIENT_ID cli-config auth.serviceaccount.id The Client ID of the service account to connect to
INSIGHTS_CLIENT_SECRET cli-config auth.serviceaccount.secret The Client Secret associated with the above client id
 cat ~/.insights/pakx-config | head -5
[envvars]
KX_PACKAGE_PATH = ""
KX_ARTIFACT_PATH = ""
KX_DEPLOYMENT_PATH = ""
KX_PACKAGE_FORMAT = ".yaml"

Default PATH values

The *_PATH variables will default to /tmp directory locations if not explicitly set. This will ensure functionality but will not guarantee state.

Also note that when choosing a path for each we recommend the following constraint:

KX_PACKAGE_PATH != KX_ARTIFACT_PATH != KX_DEPLOYMENT_PATH

Changing KX_PACKAGE_FORMAT

We do not advise changing KX_PACKAGE_FORMAT.

If it must be done, please run the below command on any packages that you are still working on in order to avoid any corruption of package data:

kxi package convert my-pkg --fmt <TARGET_FMT>
 cat ~/.insights/cli-config
[default]
hostname = https://my-insights-deployment.com
auth.serviceaccount.id = my-client-id
auth.serviceaccount.secret = my-secret

Checking a remote connection

What hostname or client should I use ?

If you have not yet set up kdb Insights Enterprise or do not have access the deployment you will need to review (one of) the below:

If you have not got a client ID and secret, you can follow the below guide:

  • Adding a user (grant theinsights.package.* role rather than the one in the guide)

After configuring the tool to connect to a running kdb Insights Enterprise instance you can verify the connection by running the below:

 kxi package remote-list
{}

This command should only take a second or two to execute and will return all packages available in you kdb Insights Enterprise deployment. In this case, there are no packages on the target system.

Debugging and logging

As well as being able to pass in the various directory variables as command line arguments (see kxi package --help we can set e.g --pkg-lib), various debug and log level flags can be passed in too.

Log level

Simply increase log verbosity by adding -v it can be added multiple times:

kxi package -vvv <subcommand>

Debug mode

Debug mode mainly controls whether the error traceback will get printed and is intended to be helpful for developer (or bug reports):

kxi package --debug <subcommand>
kxi --debug package <subcommand>

Quiet mode

Quiet mode minimises the printout to only the result without any other messages - this can be helpful if you need to pipe the output:

kxi package -q <subcommand>

Debug and log level flags are top level

These flags are passed into the package manager command before the subcommand. This is important: --debug cannot be tacked on at the end.

Next steps