Remote interaction with kdb Insights Enterprise packages
The CLI can interact with packages in kdb Insights Enterprise in the following ways:
- Pack to create a
kxi
package. This is required before you can upload it to kdb Insights Enterprise. - Upload to
push
thekxi
package to your desired kdb Insights Enterprise cluster. - List packages in your desired kdb Insights Enterprise cluster.
- Download packages from your kdb Insights Enterprise cluster to edit them locally.
- Remove packages from your kdb Insights Enterprise cluster.
.kxi
package files are known by the command line as artifacts
Packing
Before a package can be uploaded to a kdb Insights Enterprise instance you need to create a kxi
package file out of the package, which is stored in the KX_ARTIFACT_PATH
. This is facilitated using the command kxi package packit
which is defined as follows:
kxi package packit --help
Usage: kxi package packit [OPTIONS] SOURCE
Create a package artifact given a source code directory.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --override-deps TEXT Specify which deps to override during packit. │
│ format 'name/ver:path/to/new/package' │
│ --keep-unlocked Keep the original unlocked q files once the locked │
│ versions are created. │
│ --lock-q-files Lock the contents of the q files in the package. │
│ Files whose first line is /dnc or //dnc will be │
│ ignored. │
│ --all-deps Package all dependencies together. │
│ --tag Tag for release, omit the random dev hash suffix. │
│ --version TEXT Override the version of the package. │
│ --package-name TEXT Override the auto-generated package name. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
Customization
This section provides some additional information on each of the options available when using kxi package packit
, allowing you to customize the creation of a .kxi
package file locally or within your Continuous Integration (CI) environment.
The naming and versioning of packages is by default determined by the contents of the manifest
file. For many use-cases this is sufficient to allow packages to be generated, however, when automating this within Continuous Integration/Continuous Deployment (CI/CD) processes, this limitation can make the process of updating versions to align with release .kxi
package files difficult. The use of --version
enables you to take the tag version of the code repository defining your package as the version to be used when creating a .kxi
package file. Similarly, --tag
provides the ability to determine what is a production
version and what is a development
version when generating your package. The following examples outline the difference when generating a package with and without the --tag
option:
kxi package packit mypkg
Refreshing package before packing...
Creating package from mypkg
Package created: /tmp/artifact_store/mypkg-0.0.2.dev1.kxi
mypkg-0.0.2.dev1.kxi
kxi package packit mypkg --tag
Refreshing package before packing...
Creating package from mypkg
Package created: /tmp/artifact_store/mypkg-0.0.2.dev1.kxi
mypkg-0.0.2.dev1.kxi
Including --tag
will omit the random dev_hash
which is included by default. This is useful within the context of a CI/CD workflow as it enables the same package to be generated within the testing and distribution phases of your workflow without requiring a change to the underlying code.
Usage of the --tag
, --package-name
and --version
options results in changes to the manifest
file contained within a .kxi
package file. Specifically, your manifest
file is updated to reflect the name and version given during the packit
phase of your workflow.
Including all dependencies
By default, creating a .kxi
package file will not include any dependencies.
This means, to make use of your dependencies in kdb Insights Enterprise, you should either:
- Ensure all your dependencies are uploaded prior to uploading a package that depends on them.
- Ensure all packages can be remotely sourced from your kdb Insights Enterprise cluster.
- Bundle all packages together and bulk install them.
The --all-deps
flag bundles and installs all packages. kdb Insights Enterprise sources all dependencies during .kxi
package file creation and includes them all in the .kxi
files.
This functionality can be useful but does increase the size of your .kxi
package files, which now include all dependent packages.
Overriding dependencies
The override-deps
flag is an advanced feature and allows you to "swap-out" dependencies for other dependencies - it is intended to be used in development to test out if a new version of a dependency would be compatible with the package.
Other constraints when using override-deps
-
It must be run in conjunction with
--all-deps
or it will have no effect. This is because--all-deps
builds the full dependency tree. -
It cannot be used with
--tag
as we don't want to automatically--tag
a release version when testing new versions of dependencies. -
The overriding dep must be available locally
In the example below mypkg
is created and pkg-a
and pkg-b
as included as dependencies. However, when packing the package override-deps
is used to replace pkg-b
with pkg-c
.
kxi package init mypkg --force
kxi package init pkg-a --force
kxi package init pkg-b --force
kxi package init pkg-c --force
kxi package add --to mypkg dep --name pkg-a --version 0.0.1
kxi package add --to mypkg dep --name pkg-b --version 0.0.1
echo "----Setup Finished---"
artifact=$(kxi package packit mypkg --all-deps --override-deps pkg-b/0.0.1:pkg-c/0.0.1 2>&1 | tail -1)
kxi package info $artifact
Creating package at location: mypkg
Writing mypkg/manifest.yaml
Creating package at location: pkg-a
Writing pkg-a/manifest.yaml
Creating package at location: pkg-b
Writing pkg-b/manifest.yaml
Creating package at location: pkg-c
Writing pkg-c/manifest.yaml
Writing mypkg/manifest.yaml
Writing mypkg/manifest.yaml
----Setup Finished---
==PATH==
/tmp/artifact_store/mypkg-0.0.2.dev1.kxi
==OBJECT==
Artifact
==Manifest==
uuid: 14637667-9bc0-4e64-9548-feb2d7adc2e5
name: mypkg
version: 0.0.2.dev1
dependencies: []
metadata:
description: ''
authors:
- {}
entrypoints:
default: init.q
system:
_pakx_version: 1.11.0
tables: {}
databases: {}
pipelines: {}
views: {}
patches: []
Locking code
In certain circumstances, such as the dissemination of sensitive IP, it may be useful for you to be able to lock q code scripts. This process removes the ability to read the content of files and creates a locked
representation of functions, which removes the ability to display the function content.
Locking requires PyKX
To lock code, you must install PyKX
and it must be operating in 'licensed' mode.
The following examples show how to lock all code and either keep or remove the original copies of the q files:
kxi package packit mypkg --tag --lock-q-files
kxi package packit mypkg --tag --lock-q-files --keep-unlocked
Standalone lock
command for q functions.
This offers the same functionality as packit --lock-q-code
but operates directly on the package directory and does not produce an .kxi
package file.
kxi package lock --help
Usage: kxi package lock [OPTIONS] SOURCE
Lock the q files within a source directory. Files whose first line is /dnc or
//dnc will be ignored.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --keep-unlocked Keep the original unlocked q files once the locked │
│ versions are created. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
Selectively locking q files
By putting /dnc
or //dnc
at the top of a q file, you can prevent the file from being locked, even when the lock
or packit --lock-q-files
commands are run.
This is useful for keeping public functions open so they can be extended easily, while locking any private libraries that are used within them.
Uploading
Run the push
command to upload the .kxi
package file to your desired kdb Insights Enterprise deployment and install it. You can then view and deploy it in the UI or via the CLI.
To authenticate to your chosen kdb Insights Enterprise deployment, follow the instructions on Command Line Interface for Packaging.
kxi package push --help
Usage: kxi package push [OPTIONS] ARTIFACT
Publish an artifact to the running insights package manager service. Where
ARTIFACT is either the filepath or the 'name/version' of the artifact.
╭─ Authentication option overrides ────────────────────────────────────────────╮
│ --hostname,--url TEXT Insights URL │
│ --realm TEXT Realm │
│ --client-id TEXT Client id │
│ --client-secret TEXT Client secret │
│ --auth-enabled/--auth-disabled Retrieve Bearer Token │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --force Force push: overwrite existing packages and their │
│ dependencies. │
│ --server-timeout TEXT Timeout for Insights server calls │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
When you push
a package, it doesn't immediately start running. The packages becomes available for use in kdb Insights Enterprise.
Packages uploaded to kdb Insights Enterprise must be defined as a "unique name plus version" pair. If you attempt to upload a version of a package that already exists on your kdb Insights Enterprise installation, the installation is skipped, resulting in a message similar to the following:
{
"test_pkg": [
{
"version": "0.0.1",
"_status": "InstallationStatus.SKIPPED"
}
]
}
Error: All installations skipped
In the case that you are uploading a package and require packages to be overwritten you can use the --force
option as follows:
kxi package push --force test_pkg/0.0.1
{
"test_pkg": [
{
"version": "0.0.1",
"_status": "InstallationStatus.SUCCESS"
}
]
}
Listing
Once a package has been uploaded to kdb Insights Enterprise it is useful to validate it has been successfully uploaded to the server. This can be done using the remote-list
command defined as follows:
kxi package remote-list --help
Usage: kxi package remote-list [OPTIONS]
List all installed packages or artifacts on the running insights package
manager service.
╭─ Authentication option overrides ────────────────────────────────────────────╮
│ --hostname,--url TEXT Insights URL │
│ --realm TEXT Realm │
│ --client-id TEXT Client id │
│ --client-secret TEXT Client secret │
│ --auth-enabled/--auth-disabled Retrieve Bearer Token │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --fields TEXT List only the specified │
│ fields. │
│ --filter -f TEXT Key value pair to filter │
│ on. In particular name and │
│ version. │
│ --deployments If passed, list available │
│ deployments on the system. │
│ --artifacts If passed, list available │
│ artifacts on the system. │
│ --simple Simplify the output to │
│ 'name/version' format. │
│ --obj-type [Package|Artifact|Stream|D List this obj type (e.g. │
│ atabase|Pipeline|Schema|As Pipeline) overrides │
│ sembly|View] artifact/deployment flag. │
│ Case Sensitive. │
│ --output-format -o [json|simple|table] Output format for the list │
│ command │
│ --server-timeout TEXT Timeout for Insights │
│ server calls │
│ --help Show this message and │
│ exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
The following examples show listing the packages on a kdb Insights Enterprise installation containing two packages ml
and test_pkg
under various conditions:
Default list functionality:
$ kxi package remote-list
{
"test_pkg": [
{
"version": "0.0.1",
"fields": {}
}
],
"ml": [
{
"version": "1.0.0",
"fields": {}
}
]
}
Retrieving all information about the packages uploaded using the --fields
command:
$ kxi package remote-list --fields all
{
"test_pkg": [
{
"version": "0.0.1",
"fields": {
"name": "test_pkg",
"version": "0.0.1",
"license": "",
"dependencies": {},
"entrypoints": {
"default": "init.q"
},
"metadata": {
"description": "",
"authors": {
"conormccarthy": {
"email": ""
}
},
"entitlements": {},
"uuid": "fadf2c54-acaa-4a44-b948-0eb10cf2edde",
"pakxtime": "2022-12-31T19:40:13.470337"
},
"system": {
"_pakx_version": "1.2.0rc2"
},
"pipelines": {},
"udfs": {
"names": [
"udf"
]
},
"databases": {}
}
}
],
"ml": [
{
"version": "1.0.0",
"fields": {
"name": "ml",
"version": "1.0.0",
"license": "",
"dependencies": {},
"entrypoints": {
"default": "ml.q"
},
"metadata": {
"description": "lorem ipsum dolar est",
"authors": {
"conormccarthy": {
"email": null
}
},
"uuid": "15122015-75e8-4249-9d94-188d4a47d55c",
"pakxtime": "2022-12-28T14:17:24.795340"
},
"entitlements": {},
"system": {
"_pakx_version": "1.2.0rc2"
},
"pipelines": {},
"databases": {},
"udfs": {
"names": [
"udf"
]
}
}
}
]
}
Filtering on a specific named package with limited field information returned:
$ kxi package remote-list --filter name=test_pkg --fields metadata,system
{
"test_pkg": [
{
"version": "0.0.1",
"fields": {
"metadata": {
"description": "",
"authors": {
"conormccarthy": {
"email": ""
}
},
"entitlements": {},
"uuid": "fadf2c54-acaa-4a44-b948-0eb10cf2edde",
"pakxtime": "2022-12-31T19:40:13.470337"
},
"system": {
"_pakx_version": "1.2.0rc2"
}
}
}
]
}
Downloading
You can pull packages from kdb Insights Enterprise to your local machine using the pull
command. This downloads the .kxi
file to your current working directory. For example:
kxi package pull test_pkg/0.0.1
This can be very useful in conjunction with the unpack
command:
kxi package unpack test_pkg-0.0.1.kxi
To open up and validate the contents of the package. The new directory structure will be in the current directory alongside the .kxi
package file: test_pkg/
.
Tidy up...
After pulling a package, it can be beneficial to either delete the package or install it into the local package folder.
Doing either of these ensures that the package is resolved first you run any kxi package
commands.
Deleting
Once you are finished using a package in kdb Insights Enterprise you can remove it as follows:
kxi package remote-remove qpackage/1.0.0 --force
{
"qpackage": [
{
"_status": "DELETED",
"version": "1.0.0"
}
]
}
By default remote-remove
removes the packages but not the .kxi
package files. In order to clear the .kxi
package files as well, run the following command:
kxi package remote-remove --artifacts qpackage/1.0.0 --force
{
"qpackage": [
{
"_status": "DELETED",
"version": "1.0.0"
}
]
}
You can also remove your locally stored packages and .kxi
package files.
kxi package uninstall qpackage/1.0.0 --force && kxi package uninstall --artifacts qpackage/1.0.0 --force
{}
{}