Skip to content

Pushing, pulling and listing

Push

In order to get analytics and code on a kdb Insights Enterprise deployment, you need to create a kxi artifact and then push it to your desired kdb Insights Enterprise cluster.

When you push a package, it does not immediately start running, it is simply just available for importing or deploying.

kxi package push --help
Usage: kxi package push [OPTIONS] NAMEVER

  Publish an artifact to the running insights package manager service. Where
  NAMEVER is the 'name/version' of the artifact we want to push from the
  service.

Options:
  --force                         Force push: overwrite existing packages and
                                  their dependencies.
  --url TEXT                      Insights URL[env var: INSIGHTS_URL; default:
                                  https://replace.me]
  --realm TEXT                    Realm[env var: INSIGHTS_REALM; default:
                                  insights]
  --client-id TEXT                Client id[env var: INSIGHTS_CLIENT_ID;
                                  default: test-publisher]
  --client-secret TEXT            Client secret[env var:
                                  INSIGHTS_CLIENT_SECRET; default: special-
                                  secret]
  --auth-enabled / --auth-disabled
                                  Will attempt to retrieve bearer token for
                                  request  [env var: KXI_AUTH_ENABLED]
  --help                          Show this message and exit.
Setting up credentials

This has been covered in the introduction

To facilitate pushing your package to kdb Insights Enterprise you must supply the following information:

required input description CLI based input environment variable
Insights URL The URL defining the kdb Insights Enterprise instance with which you are interacting --url INSIGHTS_URL
Insights Client ID The public client identifier defining who is interacting with the kdb Insights Enterprise instance --client-id INSIGHTS_CLIENT_ID
Insights Client Secret The authorization secret associated with the public client identifier --client-secret INSIGHTS_CLIENT_SECRET

The information required for setting this configuration can be retrieved through use of following Kubernetes command assuming you have been added at install time (i.e. not added once kdb Insights Enterprise is running).

kubectl get secret keycloak-realm -n <user-namespace> -o json | jq -r ".data[]" |base64 -d

Once executed, you must find a serviceAccountClientID which has the realmRole insights.package.* use the value for serviceAccountClientID to define your auth.serviceaccount.id and search the available clients to find its corresponding secret value.

Assumption

In all the examples below unless otherwise specified it is assumed that the commands executed are using environment variable set configuration.

Once you have access to the required configuration and have decided how you will set it for your use-case, you are then in a position to upload your package.

Once a named and versioned package artifact has been created using the packit command as demonstrated here, you can upload it to kdb Insights Enterprise as follows using a named or default profile:

$ export INSIGHTS_URL="https://testnamespace.location.kx.com"
$ export INSIGHTS_CLIENT_ID="testclientid"
$ export INSIGHTS_CLIENT_SECRET="testclientsecret"
$ kxi package push test_pkg/0.0.1
$ kxi package push \
--url "https://testnamespace.location.kx.com" \
--client-id "testclientid" \
--client-secret "testclientsecret" \
test_pkg/0.0.1

Packages uploaded to kdb Insights Enterprise must be defined as a unique name + version pair, specifically, should you attempt to upload a version of a package that already exists on your kdb Insights Enterprise installation, the install is skipped, resulting in a message similar to the following:

{
    "test_pkg": [
        {
            "version": "0.0.1",
            "_status": "InstallationStatus.SKIPPED"
        }
    ]
}
Error: All installations skipped

In the case that you are uploading a package and require packages to be overwritten you can use the --force option as follows:

kxi package push --force test_pkg/0.0.1
{
    "test_pkg": [
        {
            "version": "0.0.1",
            "_status": "InstallationStatus.SUCCESS"
        }
    ]
}

List

Once a package has been pushed to kdb Insights Enterprise it may be useful to validate it has been successfully uploaded to the server. This can be done using the remote-list command defined as follows:

kxi package remote-list --help
Usage: kxi package remote-list [OPTIONS]

  List all installed packages or artifacts on the running insights package
  manager service.

Options:
  --fields TEXT                   List only the specified fields.
  -f, --filter TEXT               Key value pair to filter on. In particular
                                  name and version.
  --deployments                   If passed, list available deployments on the
                                  system.
  --artifacts                     If passed, list available artifacts on the
                                  system.
  --simple                        Simplify the output to 'name/version'
                                  format.
  --obj-type [Package|Artifact|Deployment|Stream|Database|Pipeline|Schema|Assembly]
                                  List this obj type (e.g. Pipeline) overrides
                                  artifact/deployment flag. Case Sensitive.
  --url TEXT                      Insights URL[env var: INSIGHTS_URL; default:
                                  https://replace.me]
  --realm TEXT                    Realm[env var: INSIGHTS_REALM; default:
                                  insights]
  --client-id TEXT                Client id[env var: INSIGHTS_CLIENT_ID;
                                  default: test-publisher]
  --client-secret TEXT            Client secret[env var:
                                  INSIGHTS_CLIENT_SECRET; default: special-
                                  secret]
  --auth-enabled / --auth-disabled
                                  Will attempt to retrieve bearer token for
                                  request  [env var: KXI_AUTH_ENABLED]
  --help                          Show this message and exit.

The following example shows listing the packages on a kdb Insights Enterprise installation containing two packages ml and test_pkg under various conditions:

  1. Default list functionality
  2. Retrieving all information about the packages uploaded using the --fields command
  3. Filtering on a specific named package with limited field information returned
$ kxi package remote-list
{
    "test_pkg": [
        {
            "version": "0.0.1",
            "fields": {}
        }
    ],
    "ml": [
        {
            "version": "1.0.0",
            "fields": {}
        }
    ]
}
$ kxi package remote-list --fields all
{
    "test_pkg": [
        {
            "version": "0.0.1",
            "fields": {
                "name": "test_pkg",
                "version": "0.0.1",
                "license": "",
                "dependencies": {},
                "entrypoints": {
                    "default": "init.q"
                },
                "metadata": {
                    "description": "",
                    "authors": {
                        "conormccarthy": {
                            "email": ""
                        }
                    },
                    "entitlements": {},
                    "uuid": "fadf2c54-acaa-4a44-b948-0eb10cf2edde",
                    "pakxtime": "2022-12-31T19:40:13.470337"
                },
                "system": {
                    "_pakx_version": "1.2.0rc2"
                },
                "pipelines": {},
                "udfs": {
                    "names": [
                        "udf"
                    ]
                },
                "databases": {}
            }
        }
    ],
    "ml": [
        {
            "version": "1.0.0",
            "fields": {
                "name": "ml",
                "version": "1.0.0",
                "license": "",
                "dependencies": {},
                "entrypoints": {
                    "default": "ml.q"
                },
                "metadata": {
                    "description": "lorem ipsum dolar est",
                    "authors": {
                        "conormccarthy": {
                            "email": null
                        }
                    },
                    "uuid": "15122015-75e8-4249-9d94-188d4a47d55c",
                    "pakxtime": "2022-12-28T14:17:24.795340"
                },
                "entitlements": {},
                "system": {
                    "_pakx_version": "1.2.0rc2"
                },
                "pipelines": {},
                "databases": {},
                "udfs": {
                    "names": [
                        "udf"
                    ]
                }
            }
        }
    ]
}
$ kxi package remote-list --filter name=test_pkg --fields metadata,system
{
    "test_pkg": [
        {
            "version": "0.0.1",
            "fields": {
                "metadata": {
                    "description": "",
                    "authors": {
                        "conormccarthy": {
                            "email": ""
                        }
                    },
                    "entitlements": {},
                    "uuid": "fadf2c54-acaa-4a44-b948-0eb10cf2edde",
                    "pakxtime": "2022-12-31T19:40:13.470337"
                },
                "system": {
                    "_pakx_version": "1.2.0rc2"
                }
            }
        }
    ]
}

Pull

You can pull packages from_kdb Insights Enterprise_ using the pull command (the .kxi file will be downloaded to your current working directory):

kxi package pull test_pkg/0.0.1

This can be very useful in conjunction with the unpack command:

kxi package unpack test_pkg-0.0.1.kxi

To open up and validate the contents of the package. The new directory structure will be in the current directory alongside the artifact: test_pkg/.

Tidy up...

After pulling a package and taking a look at it, it is often beneficial to either delete the package or install it (if that is required). The reason being, is that it will now be first in line to be resolved when you run any kxi package commands as per the resolution logic.