Pushing, Pulling, Listing
Before Continuing...
Make sure that you've:
Push
In order to get analytics and code on the Insights platform, we need to create a kxi artifact and then push it to our desired Insights cluster.
When we push a pckage, it does not immediately start running, it is simply just availableon the platform for importing or deploying.
kxi package push --help
Usage: kxi package push [OPTIONS] NAMEVER
Publish an artifact to the running insights package manager service. Where
NAMEVER is the 'name/version' of the artifact we want to push from the
service.
Options:
--force Force push: overwrite existing packages and
their dependencies.
--url TEXT Insights URL[env var: INSIGHTS_URL; default:
https://replace.me]
--realm TEXT Realm[env var: INSIGHTS_REALM; default:
insights]
--client-id TEXT Client id[env var: INSIGHTS_CLIENT_ID;
default: test-publisher]
--client-secret TEXT Client secret[env var:
INSIGHTS_CLIENT_SECRET; default: special-
secret]
--auth-enabled / --auth-disabled
Will attempt to retrieve bearer token for
request [env var: KXI_AUTH_ENABLED]
--help Show this message and exit.
Setting up credentials
This has been covered in Introduction
To facilitate pushing your package to kdb Insights Enterprise the users must supply the following information:
| Required input | description | CLI based input | Environment variable |
|---|---|---|---|
| Insights URL | The URL defining the insights instance with which you are interacting | --url |
INSIGHTS_URL |
| Insights Client ID | The public client identifier defining who is interacting with the Insights instance | --client-id |
INSIGHTS_CLIENT_ID |
| Insights Client Secret | The authorization secret associated with the public client identifier | --client-secret |
INSIGHTS_CLIENT_SECRET |
The information required for setting this configuration can be retrieved through use of following kubernetes command assuming your user has been added at install time (i.e. not added once insights is running).
kubectl get secret keycloak-realm -n <user-namespace> -o json | jq -r ".data[]" |base64 -d
Once executed, you must find a serviceAccountClientID which has the realmRole insights.package.* use the value for serviceAccountClientID to define your client.id and search the available clients to find its corresponding secret value.
Note
In all the examples below unless otherwise specified it is assumed that the commands executed are using environment variable set configuration.
Once you have access to the required configuration and have decided how you will set it for your use-case, you should then be in a position to upload your package.
Once a named and versioned package artifact has been created using the packit command as demonstrated here, you can upload it to kdb Insights Enterprise as follows using a named or default profile.
$ export INSIGHTS_URL="https://testnamespace.location.kx.com"
$ export INSIGHTS_CLIENT_ID="testclientid"
$ export INSIGHTS_CLIENT_SECRET="testclientsecret"
$ kxi package push test_pkg/0.0.1
$ kxi package push \
--url "https://testnamespace.location.kx.com" \
--client-id "testclientid" \
--client-secret "testclientsecret" \
test_pkg/0.0.1
Packages uploaded to kdb Insights Enterprise must be defined as a unique name + version pair, specifically, should you attempt to upload a version of a package that already exists on your kdb Insights Enterprise installation, the install is skipped, resulting in a message similar to the following:
{
"test_pkg": [
{
"version": "0.0.1",
"_status": "InstallationStatus.SKIPPED"
}
]
}
Error: All installations skipped
In the case that you are uploading a package and require packages to be overwritten you can use the --force option as follows:
kxi package push --force test_pkg/0.0.1
{
"test_pkg": [
{
"version": "0.0.1",
"_status": "InstallationStatus.SUCCESS"
}
]
}
List
Once a package has been pushed to kdb Insights Enterprise it may be useful to validate it has been successfully uploaded to the server. This can be done using the remote-list command defined as follows:
kxi package remote-list --help
Usage: kxi package remote-list [OPTIONS]
List all installed packages or artifacts on the running insights package
manager service.
Options:
--fields TEXT List only the specified fields.
-f, --filter TEXT Key value pair to filter on. In particular
name and version.
--deployments If passed, list available deployments on the
system.
--artifacts If passed, list available artifacts on the
system.
--simple Simplify the output to 'name/version'
format.
--obj-type [Package|Artifact|Deployment|Stream|Database|Pipeline|Schema|Assembly]
List this obj type (e.g. Pipeline) overrides
artifact/deployment flag. Case Sensitive.
--url TEXT Insights URL[env var: INSIGHTS_URL; default:
https://replace.me]
--realm TEXT Realm[env var: INSIGHTS_REALM; default:
insights]
--client-id TEXT Client id[env var: INSIGHTS_CLIENT_ID;
default: test-publisher]
--client-secret TEXT Client secret[env var:
INSIGHTS_CLIENT_SECRET; default: special-
secret]
--auth-enabled / --auth-disabled
Will attempt to retrieve bearer token for
request [env var: KXI_AUTH_ENABLED]
--help Show this message and exit.
The following example shows listing the packages on a kdb Insights Enterprise installation containing two packages ml and test_pkg under various conditions:
- Default list functionality
- Retrieving all information about the packages uploaded using the
--fieldscommand - Filtering on a specific named package with limited field information returned
$ kxi package remote-list
{
"test_pkg": [
{
"version": "0.0.1",
"fields": {}
}
],
"ml": [
{
"version": "1.0.0",
"fields": {}
}
]
}
$ kxi package remote-list --fields all
{
"test_pkg": [
{
"version": "0.0.1",
"fields": {
"name": "test_pkg",
"version": "0.0.1",
"license": "",
"dependencies": {},
"entrypoints": {
"default": "init.q"
},
"metadata": {
"description": "",
"authors": {
"conormccarthy": {
"email": ""
}
},
"entitlements": {},
"uuid": "fadf2c54-acaa-4a44-b948-0eb10cf2edde",
"pakxtime": "2022-12-31T19:40:13.470337"
},
"system": {
"_pakx_version": "1.2.0rc2"
},
"pipelines": {},
"udfs": {
"names": [
"udf"
]
},
"databases": {}
}
}
],
"ml": [
{
"version": "1.0.0",
"fields": {
"name": "ml",
"version": "1.0.0",
"license": "",
"dependencies": {},
"entrypoints": {
"default": "ml.q"
},
"metadata": {
"description": "lorem ipsum dolar est",
"authors": {
"conormccarthy": {
"email": null
}
},
"uuid": "15122015-75e8-4249-9d94-188d4a47d55c",
"pakxtime": "2022-12-28T14:17:24.795340"
},
"entitlements": {},
"system": {
"_pakx_version": "1.2.0rc2"
},
"pipelines": {},
"databases": {},
"udfs": {
"names": [
"udf"
]
}
}
}
]
}
$ kxi package remote-list --filter name=test_pkg --fields metadata,system
{
"test_pkg": [
{
"version": "0.0.1",
"fields": {
"metadata": {
"description": "",
"authors": {
"conormccarthy": {
"email": ""
}
},
"entitlements": {},
"uuid": "fadf2c54-acaa-4a44-b948-0eb10cf2edde",
"pakxtime": "2022-12-31T19:40:13.470337"
},
"system": {
"_pakx_version": "1.2.0rc2"
}
}
}
]
}
Pull
You can pull packages on insights platform using the pull command (the .kxi file will be downloaded to your current working directory):
kxi package pull test_pkg/0.0.1
This can be very handy in conjunction with the unpack command:
kxi package unpack test_pkg-0.0.1.kxi
To open up and validate the contents of the package. The new directory structure will be in the current directory along side the artifact: test_pkg/.
Tidy up...
After pulling a package and taking a look at it, it's often beneficial to either delete the package or install it (if that's required).
The reason being, is that it will now be first in line to be resolved when you run any kxi package commands as per the resolution logic