Skip to content

Command Line Interface for Packaging

A package management tool as part of the kdb Insights CLI which should be used to manage packages from development to deployment. Where possible, use the tool to make changes to your package as the tool ensures everything is validated and correctly formatted.

The kdb Insights CLI allows you to create and modify your packages locally and upload them to kdb Insights Enterprise for deployment.

For information on installing the kdb Insights CLI, refer to Install.

Configuration

The CLI can act on files in any of the following locations:

  • The current working folder.
  • The package path (KX_PACKAGE_PATH) - packages in this location are known to be installed locally.
  • The .kxi package path (KX_ARTIFACT_PATH) - the location where .kxi packages are stored. Before uploading to kdb Insights Enterprise, packages need to be packed as .kxi package files.
  • The kdb Insights Enterprise URL - packages can be uploaded, deployed, torn down and removed from kdb Insights Enterprise. For more information, refer to the kdb Insights Enterprise connection section.

You can specify the packaging specific variables as environment variables or in the ~/.insights/pakx-config file as defined below:

variable file key in config default description
KX_PACKAGE_PATH pakx-config KX_PACKAGE_PATH /tmp Where "installed" packages are stored
KX_ARTIFACT_PATH pakx-config KX_ARTIFACT_PATH /tmp Where .kxi packages are stored

Environment variables take precedence over keys in the configuration file. For example: export KX_PACKAGE_PATH=/some/where.

Recommendations when setting these variables.

  • If the KX_PACKAGE_PATH is not set, it will be set to a temporary directory (which your OS may clean up unexpectedly).

  • KX_PACKAGE_PATH and KX_ARTIFACT_PATH should not be set to the same folder.

❯ cat ~/.insights/pakx-config | head -5
[envvars]
KX_PACKAGE_PATH = ""
KX_ARTIFACT_PATH = ""
KX_PACKAGE_FORMAT = ".yaml"

kxi package help

The kxi package command offers a wide range of tools to build, manage and deploy packages to kdb Insights Enterprise. Run the command with the --help modifier to access detailed helper documentation. The following example is a condensed version of the command:

kxi package --help 

 Usage: kxi package [OPTIONS] COMMAND [ARGS]...                                 

 KX Package Import/Export CLI.                                                  

╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --version                            Show the version and exit.              │
│ --artifact-store      DIRECTORY                                              │
│ --pkg-lib             DIRECTORY                                              │
│ --debug                              Enable stacktrace print statements and  │
│                                      increase verbosity for easier           │
│                                      debugging.                              │
│ --verbose         -v  INTEGER RANGE  Increase verbosity level                │
│ --quiet           -q                 Disable logs. Only print errors.        │
│ --stacktrace                         Enable stacktrace print statements.     │
│ --help                               Show this message and exit.             │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ When pulling/pushing to a kdb Insights Enterprise package repository ───────╮
│ pull                     Download an artifact from the running insights      │
│                          package manager service to your local directory.    │
│                          Where NAMEVER is the 'name[/version]' of the        │
│                          artifact we want to pull from the service.          │
│ push                     Publish an artifact to the running insights package │
│                          manager service. Where ARTIFACT is either the       │
│                          filepath or the 'name/version' of the artifact.     │
│ remote-list              List all installed packages or artifacts on the     │
│                          running insights package manager service.           │
│ remote-remove            Remove packages or artifacts from the running       │
│                          insights package manager service.                   │
│ remote-convert           Convert V1/V2 assemblies into Packages with the     │
│                          server. Optionally keep the package installed on    │
│                          the server.                                         │
│ remote-migration-retry   Get kxi-controller backup files.                    │
│ remote-migration-backup  Get kxi-controller backup files.                    │
│ remote-config            (Deprecated) Set a configuration option for the     │
│                          remote package manager. Use `remote config set`     │
│                          instead.                                            │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ For local development ──────────────────────────────────────────────────────╮
│ init        Creates a bare package at the specified target path.             │
│ install     Install a package given a requirement or path.                   │
│ list        List all installed packages or artifacts.                        │
│ lock        Lock the q files within a source directory. Files whose first    │
│             line is /dnc or //dnc will be ignored.                           │
│ packit      Create a package artifact given a source code directory.         │
│ uninstall   Uninstall specified locally installed packages and artifacts.    │
│ unpack      Unpack an artifact to a specified location.                      │
│ validate    Validate the contents of a source directory to ensure it is a    │
│             valid package.                                                   │
│ checkpoint  Create a package artifact given a source code directory.         │
│ convert     Converts Assembly specs to packages or changes package format.   │
│ info        show some info about the package                                 │
│ refresh     Refresh a package, picking up any available changes from disk    │
│ overlay     Overlay a package using a spec file containing a subset of       │
│             fields.                                                          │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ For managing deployments & runtime of packages on kdb Insights Enterprise ──╮
│ deploy    Deploy a package to an insights instance.                          │
│ teardown  Teardown one or multiple deployed packages running on an insights  │
│           instance                                                           │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ For managing components added to package ───────────────────────────────────╮
│ config      Local configuration options.                                     │
│ remote      V2 Remote commands. Beta.                                        │
│ add         Add an entity to the specified package.                          │
│ copy        Copy an entity from the specified package.                       │
│ rm          Remove an entity from the specified package.                     │
│ field       Find, list or mutate fields within a package.                    │
╰──────────────────────────────────────────────────────────────────────────────╯

Naming constraints

There are some constraints on naming of packages and the files and folders they contain:

  • Packages are not allowed to have an _ in their name. This is a constraint of the Kubernetes runtime where packages are deployed.

  • At the root of the package, certain file and folder names are reserved for use by the CLI and should not be used:

    • Folder names: pipelines, reports, databases, router, deployment_config, tables
    • File names: udfs.* and manifest.*

Package search paths

When searching for packages, the CLI searches paths in the following order:

  1. The package path (KX_PACKAGE_PATH)
  2. The .kxi package (KX_ARTIFACT_PATH)

In the case where you specify something such as, for example,: mypackage-0.0.1.kxi, the resolver tries to resolve a .kxi package first but falls back on the package locations as above if no .kxi package is available.

Package Versioning names

Generally we use "slash" to delimit package-name/package-version and the tool will split this up and try and resolve it based on the above search path precedence.

When dealing with a remote package, you must specify package-name/package-version using a forward slash to separate them.

This is so that the server can split the name and version and find the package on the host in order to return it.

kdb Insights Enterprise connection

After configuring the tool to connect to a running kdb Insights Enterprise instance as described in CLI authentication, you can verify the connection by running the following command:

❯ kxi package remote-list
{}

This command returns all available packages in your kdb Insights Enterprise deployment.

In the above example, there are no packages on the kdb Insights Enterprise deployment.

Debugging and logging

As well as being able to pass in the various directory variables as command line arguments, you can pass in debug and log level flags as well. Run the kxi package --help command defined in the kxi Package help section for more information and a full list of possible arguments.

Log level

To increase log verbosity, add the -v modifier to the kxi package command. You can add this modifier multiple times to increase the log level further. The following example demonstrates this:

kxi package -vvv <subcommand>

Debug mode

Debug mode controls whether the error traceback will get printed and is intended to be helpful for developers:

kxi package --debug <subcommand>

Quiet mode

Quiet mode minimises the printout to only the result without any other messages - this can be helpful if you need to pipe the output:

kxi package -q <subcommand>

Debug and log level flags need to be directly after kxi package when running the command

For example:

kxi package --debug --add databsae

Next steps