Packaging Quickstart
Packaging Quickstart
kxi package help
The kxi package command offers a wide range of tools to build, manage and even deploy packages to kdb Insights Enterprise.
It has a detailed help doc which has been collapsed below for brevity.
kxi package --help
Usage: kxi package [OPTIONS] COMMAND [ARGS]...
KX Package Import/Export CLI.
Options:
--artifact-store DIRECTORY Directory in which to store packed
artifacts;[default] $KX_ARTIFACT_PATH.[env
var: KX_ARTIFACT_PATH]
--pkg-lib DIRECTORY Directory in which packages will be
installed;[default] $KX_PACKAGE_PATH.[env var:
KX_PACKAGE_PATH]
--deployment-store DIRECTORY Directory in which deployment data is
stored;[default] $KX_DEPLOYMENT_PATH.[env var:
KX_DEPLOYMENT_PATH]
--version Version information.
--debug Enable stacktrace print statements for easier
debugging.
-v, --verbose Increase verbosity level
-q, --quiet Disable logs. Only print errors.
--help Show this message and exit.
Commands:
When pulling/pushing to a kdb Insights Enterprise package repository, use these commands:
pull Download an artifact from the running insights package
manager service to...
push Publish an artifact to the running insights package manager
service.
remote-list List all installed packages or artifacts on the running
insights package...
remote-remove Remove packages or artifacts from the running insights
package manager...
For local development, use these commands:
convert Converts Assembly specs to packages or changes package format.
info show some info about the package
init Creates a bare package at the specified target path.
install Install a package given a requirement or path.
list List all installed packages or artifacts.
lock Lock the q files within a source directory.
overlay Overlay a package using a spec file containing a subset of
fields.
packit Create a package artifact given a source code directory.
refresh Refresh a package, picking up any available changes from disk
uninstall Uninstall specified locally installed packages and artifacts.
unpack Unpack an artifact to a specified location.
validate Validate the contents of a source directory to ensure it is a
valid package.
For managing deployments & runtime of packages on kdb Insights Enterprise, use these commands:
deploy Deploy a package to an insights instance
teardown Teardown a deployed packaged running on an insights instance
For managing components added to package, use these commands:
add Add an entity to the specified package.
copy Copy an entity from the specified package.
field Find, list or mutate fields within a package.
rm Remove an entity from the specified package.
Create a Package
Before Continuing...
Make sure that you've:
Interactions with packages can be local or remote. In this section we will deal with creating a package on your local system.
By the end of this step, you should be able to...
- Initialize a package locally.
- Update this package to include user-defined code.
- Package this entity to generate an artifact.
- Install this package to your local system
Before we start we should set our package path (note this can be done in config to avoid this step).
export KX_PACKAGE_PATH=docs-packages/packages
export KX_ARTIFACT_PATH=docs-packages/artifacts
mkdir -p KX_PACKAGE_PATH
mkdir -p KX_ARTIFACT_PATH
kxi package init qpackage
Creating package at location: qpackage
Writing qpackage/manifest.yaml
info about the package below:
kxi package info qpackage
==PATH==
/builds/kxdev/kxinsights/documentation/code-kx-com/qpackage
==OBJECT==
Package
==Manifest==
name: qpackage
version: 0.0.1
metadata:
description: ''
authors:
- name: root
email: ''
entrypoints:
default: init.q
Create a file src/sp.q within the qpackage directory.
This file contains a User-Defined Function (UDF).
cd qpackage
mkdir -p src/
cat << EOF > src/sp.q
.test.variable:1b
// @udf.name("sp_map")
// @udf.tag("sp")
// @udf.category("map")
.test.sp.map:{[table;params]
select from table where x<10
}
EOF
cd ..
We can modify the init.q file which is our default entrypoint in order to ensure that our new module gets loaded by default.
cd qpackage
cat << EOF > init.q
// Load the src/sp.q file relative to package root
.kxi.packages.file.load["src/sp.q"]
EOF
cd ..
Package the artifact ensuring we define the version of the package on the command line.
What does packit mean in this context?
Packing here creates an archive (zip) of the package directory and writes it to KX_ARTIFACT_PATH
The extension of the artifact is .kxi but it behaves just like a zip
kxi package convert qpackage --fmt json
kxi package packit qpackage --version 1.0.0 --tag
Refreshing package before packing...
Writing qpackage/manifest.json
Creating package from qpackage
Package created: /tmp/artifact_store/qpackage-1.0.0.kxi
qpackage-1.0.0.kxi
Yaml vs Json
You may have noticed the convert call above.
This is necessary because most local systems don't have support for kdbyaml which does the yaml parsing in q.
We will change this in an upcoming release but for now, when running locally, it is easier to stick with json.
List the current artifacts available locally.
kxi package list --artifacts
{
"qpackage": [
{
"fields": {},
"version": "1.0.0"
}
]
}
Install the package locally such that it can be loaded into python or q processes locally.
What does Installing mean in this context?
Installing here simply means "unpacking" the .kxi object into the KX_PACKAGE_PATH.
This is so, later when we load the code, we know where to look for it!
kxi package install qpackage-1.0.0.kxi --force
qpackage-1.0.0-local
{
"qpackage": [
{
"version": "1.0.0",
"_status": "InstallationStatus.SUCCESS"
}
]
}
Do I need to packit before install?
We can skip the packit step and install our qpackage directly from our working directory:
pakx install qpackage
Congratulations, you have developed, packed and installed a package onto your system.
Run the package code locally
By the end of this step, you should be able to...
- Install the
kxi.packageslibrary - Run the code locally
In order to run our package we will first need to install the kxi import library.
pip install --extra-index-url https://nexus.dl.kx.com/repository/kxi-pypi-public/simple/ kxi.packages
Then to see our available packages we can simply run:
import kxi.packages as kxi
kxi.setup_env.set_package_path("docs-packages/packages")
kxi.init()
print(kxi.packages.list())
name versions
0 qpackage 1.0.0
q).kxi.packages.list.all[]
name versions
-----------------------
"qpackage" "1.0.0"
And in order to load and run any functions we have defined we can use the below:
import pykx as kx
kxi.packages.load("qpackage","1.0.0")
print(kx.q(".test"))
| ::
variable| 1b
sp | ``map!(::;{[table;params]
select from table where x<10
})
q).kxi.packages.load["qpackage";"1.0.0"]
q).test
variable| 1b
sp | ``map!(::;{[table;params]
select from table where x<10
})
kxi.packages.load
The kxi.packages.load function behaves like a "global" import, assigning all variables relative to the current namespace.
In python it is similar to doing from module import *.
Congratulations you've run your package code! Next we will push it to kdb Insights Enterprise and interact with it there.
Push a Package
By the end of this step, you should be able to...
- Push the code to kdb Insights Enterprise
- Run the code in kdb Insights Enterprise
Let's see what packages are currently available on the system
kxi package remote-list
{}
OK, nothing there, lets push up our new package:
kxi package push qpackage/1.0.0
{
"qpackage": [
{
"version": "1.0.0",
"_status": "InstallationStatus.SUCCESS"
}
]
}
Now, re-running the remote-list command, we can see our new package:
kxi package remote-list
{
"qpackage": [
"1.0.0"
]
}
Great, our package is on the system and the code is available for us to use!
Using Code in Insights
If we log into our kdb Insights Enterprise instance we can leverage our code in the UI.
-
List and load the content of a package within a query session.
-
Using an available UDF within an Pipeline developed within a query session.
-
Define a map node within a Pipeline to use a UDF.
Removing Packages
Once we are finished we can delete the package from the kdb Insights Enterprise instance.
kxi package remote-remove qpackage/1.0.0 --force
{
"qpackage": [
{
"_status": "DELETED",
"version": "1.0.0"
}
]
}
remote-remove
remote-remove by default removes the packages but not the Artifacts.
In order to clear the artifacts too:
kxi package remote-remove --artifacts qpackage/1.0.0 --force
{
"qpackage": [
{
"_status": "DELETED",
"version": "1.0.0"
}
]
}
We can also remove our locally stored packages and artifacts in a similar way:
kxi package uninstall qpackage/1.0.0 --force && kxi package uninstall --artifacts qpackage/1.0.0 --force
{}
{
"qpackage": [
{
"_status": "DELETED",
"version": "1.0.0"
}
]
}