Skip to content

Introduction

What is a kdb Insights Package ?

A package is simply a directory structure consisting of:

  • Config to define Deployable Insights Components (e.g. databases, pipelines)
  • Code to define User Defined Analytics to run on those components
  • A single Manifest file to keep track of components, code and dependencies

The packaging toolkit facilitates:

  • Creating and executing custom analytics
  • Deploying entities leveraging those custom analytics

Installation of the Packaging Tool

KX has produced a package management tool as part of the KXI CLI which should be used to manage packages from development to deployment. Where possible the tool should be used to make changes to your package as it will ensure everything is validated and correctly formatted.

Python, pip & virtualenv

The CLI requires python (>=3.8, <=3.10).

Once installed we recommend using a virtualenv to ensure there are no dependency clashes.

python3 -m venv my-new-pkg-env
source my-new-pkg-env/bin/activate
pip install --extra-index-url https://nexus.dl.kx.com/repository/kxi-pypi-public/simple/ kxicli

For more information on installing the CLI see: installing the KXI CLI

Configuring the tool

Checking the Installation

This section explains how to configure the tool locally for manipulation, installation and storage of package artifacts.

To check that the packaging tool is installed, run:

kxi package --version

{
    "version": "1.8.3",
    "env": {
        "KX_PACKAGE_PATH": null,
        "KX_ARTIFACT_PATH": null,
        "KXI_AUTH_ENABLED": null,
        "INSIGHTS_URL": null,
        "INSIGHTS_CLIENT_SECRET": null,
        "INSIGHTS_CLIENT_ID": null,
        "INSIGHTS_REALM": null
    }
}
Config & Env Vars Precedence

Below are some remarks around specifying values in the shell vs in the config files:

  • Each var can be explicitly set (export KX_PACKAGE_PATH=/some/where)
  • Each var can be explicitly set in its respective config file (see below table)
  • Variables set in shell will take precedence over ones set in config
Config & Env Vars Explained

A Tale of Two Configs

There are two configs that the packaging tool will read

  • ~/.insights/cli-config
  • ~/.insights/pakx-config
Env Var Config Key in config Description
KX_PACKAGE_PATH pakx-config KX_PACKAGE_PATH Where "Installed" packages will be stored.
KX_ARTIFACT_PATH pakx-config KX_ARTIFACT_PATH Where package archives or "Artifacts" will be stored.
KX_DEPLOYMENT_PATH pakx-config KX_DEPLOYMENT_PATH Where "Deployment" objects will be stored.
KX_PACKAGE_FORMAT pakx-config KX_PACKAGE_FORMAT The default package format (.yaml by default, can also be .json)
INSIGHTS_URL cli-config hostname The URL to the kdb Insights Enterprise instance to which you want to connect
INSIGHTS_REALM cli-config realm The Insights realm name (defaults to kdb Insights Enterprise)
INSIGHTS_CLIENT_ID cli-config client.id The Client ID of the service account to connect to
INSIGHTS_CLIENT_SECRET cli-config client.secret The Client Secret associated with the above client id 
❯ cat ~/.insights/pakx-config | head -5
[envvars]
KX_PACKAGE_PATH = ""
KX_ARTIFACT_PATH = ""
KX_DEPLOYMENT_PATH = ""
KX_PACKAGE_FORMAT = ".yaml"

Default PATH values

The *_PATH variables will default to /tmp directory locations if not explicitly set. This will ensure functionality but will not guarantee state.

Also note that when choosing a path for each we recommend the following constraint: 

KX_PACKAGE_PATH != KX_ARTIFACT_PATH != KX_DEPLOYMENT_PATH

Changing KX_PACKAGE_FORMAT

We do not advise changing KX_PACKAGE_FORMAT.

If it must be done, please run the below command on any packages that you are still working on in order to avoid any corruption of package data: 

kxi package convert my-pkg --fmt <TARGET_FMT>


❯ cat ~/.insights/cli-config
[default]
hostname = https://my-insights-deployment.com
client.id = my-client-id
client.secret = my-secret
Checking a Remote Connection
"What hostname or client should I use ?"

If you have not yet set up Insights or don't have access to an Insights Instance you will need to review (one of) the below:

If you have not got a client ID and secret, you can follow the below guide:

After configuring the tool to connect to a running kdb Insights Enterprise instance you can verify the connection by running the below: 

❯ kxi package remote-list
{}
This command should only take a second or two to execute and will return all packages available on Insights. In this case, there are no packages on the target system

Debugging & Logging

As well as being able to pass in the various directory variables as command line args (see kxi package --help we can set e.g --pkg-lib), various debug and log level flags can be passed in too.

Log Level

Simply increase log verbosity by adding -v it can be added multiple times: 

kxi package -vvv <subcommand>

Debug mode

Debug mode mainly controls whether or not the error traceback will get printed and is intended to be helpful for developer (or bug reports): 

kxi package --debug <subcommand>
kxi --debug package <subcommand>

Quiet mode

Quiet mode minimises the printout to only the result without any other messages - this can be handy if you need to pipe the output: 

kxi package -q <subcommand>

Debug and log level flags are top level

These flags are passed into the package manager command before the subcommand. This is important: --debug can't be tacked on at the end.

Next Steps