Skip to content

Remote interaction with kdb Insights Enterprise packages

The CLI can interact with packages in kdb Insights Enterprise in the following ways:

  1. Pack to create a kxi package. This is required before you can upload it to kdb Insights Enterprise.
  2. Upload to push the kxi package to your desired kdb Insights Enterprise cluster.
  3. List packages in your desired kdb Insights Enterprise cluster.
  4. Download packages from your kdb Insights Enterprise cluster to edit them locally.
  5. Remove packages from your kdb Insights Enterprise cluster.

.kxi package files are known by the command line as artifacts

Packing

Before a package can be uploaded to a kdb Insights Enterprise instance you need to create a kxi package file out of the package, which is stored in the KX_ARTIFACT_PATH. This is facilitated using the command kxi package packit which is defined as follows:

kxi package packit --help
 Usage: kxi package packit [OPTIONS] SOURCE                                     

 Create a package artifact given a source code directory.                       

╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --override-deps    TEXT  Specify which deps to override during packit.       │
│                          format 'name/ver:path/to/new/package'               │
│ --keep-unlocked          Keep the original unlocked q files once the locked  │
│                          versions are created.                               │
│ --lock-q-files           Lock the contents of the q files in the package.    │
│                          Files whose first line is /dnc or //dnc will be     │
│                          ignored.                                            │
│ --all-deps               Package all dependencies together.                  │
│ --tag                    Tag for release, omit the random dev hash suffix.   │
│ --version          TEXT  Override the version of the package.                │
│ --package-name     TEXT  Override the auto-generated package name.           │
│ --help                   Show this message and exit.                         │
╰──────────────────────────────────────────────────────────────────────────────╯

Customization

This section provides some additional information on each of the options available when using kxi package packit, allowing you to customize the creation of a .kxi package file locally or within your Continuous Integration (CI) environment.

The naming and versioning of packages is by default determined by the contents of the manifest file. For many use-cases this is sufficient to allow packages to be generated, however, when automating this within Continuous Integration/Continuous Deployment (CI/CD) processes, this limitation can make the process of updating versions to align with release .kxi package files difficult. The use of --version enables you to take the tag version of the code repository defining your package as the version to be used when creating a .kxi package file. Similarly, --tag provides the ability to determine what is a production version and what is a development version when generating your package. The following examples outline the difference when generating a package with and without the --tag option:

kxi package packit mypkg
Refreshing package before packing...
Creating package from mypkg
Package created: /tmp/artifact_store/mypkg-0.0.2.dev1.kxi
mypkg-0.0.2.dev1.kxi

kxi package packit mypkg --tag
Refreshing package before packing...
Creating package from mypkg
Package created: /tmp/artifact_store/mypkg-0.0.2.dev1.kxi
mypkg-0.0.2.dev1.kxi

Including --tag will omit the random dev_hash which is included by default. This is useful within the context of a CI/CD workflow as it enables the same package to be generated within the testing and distribution phases of your workflow without requiring a change to the underlying code.

Usage of the --tag, --package-name and --version options results in changes to the manifest file contained within a .kxi package file. Specifically, your manifest file is updated to reflect the name and version given during the packit phase of your workflow.

Including all dependencies

By default, creating a .kxi package file will not include any dependencies.

This means, to make use of your dependencies in kdb Insights Enterprise, you should either:

  • Ensure all your dependencies are uploaded prior to uploading a package that depends on them.
  • Ensure all packages can be remotely sourced from your kdb Insights Enterprise cluster.
  • Bundle all packages together and bulk install them.

The --all-deps flag bundles and installs all packages. kdb Insights Enterprise sources all dependencies during .kxi package file creation and includes them all in the .kxi files.

This functionality can be useful but does increase the size of your .kxi package files, which now include all dependent packages.

Overriding dependencies

The override-deps flag is an advanced feature and allows you to "swap-out" dependencies for other dependencies - it is intended to be used in development to test out if a new version of a dependency would be compatible with the package.

Other constraints when using override-deps

  • It must be run in conjunction with --all-deps or it will have no effect. This is because --all-deps builds the full dependency tree.

  • It cannot be used with --tag as we don't want to automatically --tag a release version when testing new versions of dependencies.

  • The overriding dep must be available locally

In the example below mypkg is created and pkg-a and pkg-b as included as dependencies. However, when packing the package override-deps is used to replace pkg-b with pkg-c.

kxi package init mypkg --force 
kxi package init pkg-a --force 
kxi package init pkg-b --force 
kxi package init pkg-c --force 
kxi package add --to mypkg dep --name pkg-a --version 0.0.1 
kxi package add --to mypkg dep --name pkg-b --version 0.0.1 
echo "----Setup Finished---"
artifact=$(kxi package packit mypkg --all-deps --override-deps pkg-b/0.0.1:pkg-c/0.0.1 2>&1 | tail -1)
kxi package info $artifact
Creating package at location: mypkg
Writing mypkg/manifest.yaml
Creating package at location: pkg-a
Writing pkg-a/manifest.yaml
Creating package at location: pkg-b
Writing pkg-b/manifest.yaml
Creating package at location: pkg-c
Writing pkg-c/manifest.yaml
Writing mypkg/manifest.yaml
Writing mypkg/manifest.yaml
----Setup Finished---
==PATH==
/tmp/artifact_store/mypkg-0.0.2.dev1.kxi

==OBJECT==
Artifact

==Manifest==
uuid: 14637667-9bc0-4e64-9548-feb2d7adc2e5
name: mypkg
version: 0.0.2.dev1
dependencies: []
metadata:
  description: ''
  authors:
  - {}
entrypoints:
  default: init.q
system:
  _pakx_version: 1.11.0
tables: {}
databases: {}
pipelines: {}
views: {}
patches: []

Locking code

In certain circumstances, such as the dissemination of sensitive IP, it may be useful for you to be able to lock q code scripts. This process removes the ability to read the content of files and creates a locked representation of functions, which removes the ability to display the function content.

Locking requires PyKX

To lock code, you must install PyKX and it must be operating in 'licensed' mode.

The following examples show how to lock all code and either keep or remove the original copies of the q files:

kxi package packit mypkg --tag --lock-q-files
kxi package packit mypkg --tag --lock-q-files --keep-unlocked

Standalone lock command for q functions.

This offers the same functionality as packit --lock-q-code but operates directly on the package directory and does not produce an .kxi package file.

kxi package lock --help
 Usage: kxi package lock [OPTIONS] SOURCE                                       

 Lock the q files within a source directory. Files whose first line is /dnc or  
 //dnc will be ignored.                                                         

╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --keep-unlocked      Keep the original unlocked q files once the locked      │
│                      versions are created.                                   │
│ --help               Show this message and exit.                             │
╰──────────────────────────────────────────────────────────────────────────────╯

Selectively locking q files

By putting /dnc or //dnc at the top of a q file, you can prevent the file from being locked, even when the lock or packit --lock-q-files commands are run. This is useful for keeping public functions open so they can be extended easily, while locking any private libraries that are used within them.

Uploading

Run the push command to upload the .kxi package file to your desired kdb Insights Enterprise deployment and install it. You can then view and deploy it in the UI or via the CLI.

To authenticate to your chosen kdb Insights Enterprise deployment, follow the instructions on Command Line Interface for Packaging.

kxi package push --help
 Usage: kxi package push [OPTIONS] ARTIFACT                                     

 Publish an artifact to the running insights package manager service. Where     
 ARTIFACT is either the filepath or the 'name/version' of the artifact.         

╭─ Authentication option overrides ────────────────────────────────────────────╮
│ --hostname,--url                  TEXT  Insights URL                         │
│ --realm                           TEXT  Realm                                │
│ --client-id                       TEXT  Client id                            │
│ --client-secret                   TEXT  Client secret                        │
│ --auth-enabled/--auth-disabled          Retrieve Bearer Token                │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --force                   Force push: overwrite existing packages and their  │
│                           dependencies.                                      │
│ --server-timeout    TEXT  Timeout for Insights server calls                  │
│ --help                    Show this message and exit.                        │
╰──────────────────────────────────────────────────────────────────────────────╯

When you push a package, it doesn't immediately start running. The packages becomes available for use in kdb Insights Enterprise.

Packages uploaded to kdb Insights Enterprise must be defined as a "unique name plus version" pair. If you attempt to upload a version of a package that already exists on your kdb Insights Enterprise installation, the installation is skipped, resulting in a message similar to the following:

{
    "test_pkg": [
        {
            "version": "0.0.1",
            "_status": "InstallationStatus.SKIPPED"
        }
    ]
}
Error: All installations skipped

In the case that you are uploading a package and require packages to be overwritten you can use the --force option as follows:

kxi package push --force test_pkg/0.0.1
{
    "test_pkg": [
        {
            "version": "0.0.1",
            "_status": "InstallationStatus.SUCCESS"
        }
    ]
}

Listing

Once a package has been uploaded to kdb Insights Enterprise it is useful to validate it has been successfully uploaded to the server. This can be done using the remote-list command defined as follows:

kxi package remote-list --help
 Usage: kxi package remote-list [OPTIONS]                                       

 List all installed packages or artifacts on the running insights package       
 manager service.                                                               

╭─ Authentication option overrides ────────────────────────────────────────────╮
│ --hostname,--url                  TEXT  Insights URL                         │
│ --realm                           TEXT  Realm                                │
│ --client-id                       TEXT  Client id                            │
│ --client-secret                   TEXT  Client secret                        │
│ --auth-enabled/--auth-disabled          Retrieve Bearer Token                │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --fields              TEXT                        List only the specified    │
│                                                   fields.                    │
│ --filter          -f  TEXT                        Key value pair to filter   │
│                                                   on. In particular name and │
│                                                   version.                   │
│ --deployments                                     If passed, list available  │
│                                                   deployments on the system. │
│ --artifacts                                       If passed, list available  │
│                                                   artifacts on the system.   │
│ --simple                                          Simplify the output to     │
│                                                   'name/version' format.     │
│ --obj-type            [Package|Artifact|Stream|D  List this obj type (e.g.   │
│                       atabase|Pipeline|Schema|As  Pipeline) overrides        │
│                       sembly|View]                artifact/deployment flag.  │
│                                                   Case Sensitive.            │
│ --output-format   -o  [json|simple|table]         Output format for the list │
│                                                   command                    │
│ --server-timeout      TEXT                        Timeout for Insights       │
│                                                   server calls               │
│ --help                                            Show this message and      │
│                                                   exit.                      │
╰──────────────────────────────────────────────────────────────────────────────╯

The following examples show listing the packages on a kdb Insights Enterprise installation containing two packages ml and test_pkg under various conditions:

Default list functionality:

$ kxi package remote-list
{
    "test_pkg": [
        {
            "version": "0.0.1",
            "fields": {}
        }
    ],
    "ml": [
        {
            "version": "1.0.0",
            "fields": {}
        }
    ]
}

Retrieving all information about the packages uploaded using the --fields command:

$ kxi package remote-list --fields all
{
    "test_pkg": [
        {
            "version": "0.0.1",
            "fields": {
                "name": "test_pkg",
                "version": "0.0.1",
                "license": "",
                "dependencies": {},
                "entrypoints": {
                    "default": "init.q"
                },
                "metadata": {
                    "description": "",
                    "authors": {
                        "conormccarthy": {
                            "email": ""
                        }
                    },
                    "entitlements": {},
                    "uuid": "fadf2c54-acaa-4a44-b948-0eb10cf2edde",
                    "pakxtime": "2022-12-31T19:40:13.470337"
                },
                "system": {
                    "_pakx_version": "1.2.0rc2"
                },
                "pipelines": {},
                "udfs": {
                    "names": [
                        "udf"
                    ]
                },
                "databases": {}
            }
        }
    ],
    "ml": [
        {
            "version": "1.0.0",
            "fields": {
                "name": "ml",
                "version": "1.0.0",
                "license": "",
                "dependencies": {},
                "entrypoints": {
                    "default": "ml.q"
                },
                "metadata": {
                    "description": "lorem ipsum dolar est",
                    "authors": {
                        "conormccarthy": {
                            "email": null
                        }
                    },
                    "uuid": "15122015-75e8-4249-9d94-188d4a47d55c",
                    "pakxtime": "2022-12-28T14:17:24.795340"
                },
                "entitlements": {},
                "system": {
                    "_pakx_version": "1.2.0rc2"
                },
                "pipelines": {},
                "databases": {},
                "udfs": {
                    "names": [
                        "udf"
                    ]
                }
            }
        }
    ]
}

Filtering on a specific named package with limited field information returned:

$ kxi package remote-list --filter name=test_pkg --fields metadata,system
{
    "test_pkg": [
        {
            "version": "0.0.1",
            "fields": {
                "metadata": {
                    "description": "",
                    "authors": {
                        "conormccarthy": {
                            "email": ""
                        }
                    },
                    "entitlements": {},
                    "uuid": "fadf2c54-acaa-4a44-b948-0eb10cf2edde",
                    "pakxtime": "2022-12-31T19:40:13.470337"
                },
                "system": {
                    "_pakx_version": "1.2.0rc2"
                }
            }
        }
    ]
}

Downloading

You can pull packages from kdb Insights Enterprise to your local machine using the pull command. This downloads the .kxi file to your current working directory. For example:

kxi package pull test_pkg/0.0.1

This can be very useful in conjunction with the unpack command:

kxi package unpack test_pkg-0.0.1.kxi

To open up and validate the contents of the package. The new directory structure will be in the current directory alongside the .kxi package file: test_pkg/.

Tidy up...

After pulling a package, it can be beneficial to either delete the package or install it into the local package folder. Doing either of these ensures that the package is resolved first you run any kxi package commands.

Deleting

Once you are finished using a package in kdb Insights Enterprise you can remove it as follows:

kxi package remote-remove qpackage/1.0.0 --force
{
    "qpackage": [
        {
            "_status": "DELETED",
            "version": "1.0.0"
        }
    ]
}

By default remote-remove removes the packages but not the .kxi package files. In order to clear the .kxi package files as well, run the following command:

kxi package remote-remove --artifacts qpackage/1.0.0 --force
{
    "qpackage": [
        {
            "_status": "DELETED",
            "version": "1.0.0"
        }
    ]
}

You can also remove your locally stored packages and .kxi package files.

kxi package uninstall qpackage/1.0.0 --force && kxi package uninstall --artifacts qpackage/1.0.0 --force
{}
{}

Next steps