Pack a Package
This page describes how to pack a package to a single sharable .kxi in kdb Insights Enterprise.
Before uploading a package to a kdb Insights Enterprise instance, you can optionally convert it to a .kxi package file, which is stored in the KX_ARTIFACT_PATH. This is useful for sharing the package with other users or for other distribution purposes.
To create the .kxi package file,use the kxi package packit command. This command allows you to customize the creation of a .kxi package file either locally or within your Continuous Integration (CI) environment. It is defined as follows:
kxi package packit --help
Usage: kxi package packit [OPTIONS] SOURCE
Create a package artifact given a source code directory.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --override-deps TEXT Specify which deps to override during packit. │
│ format 'name/ver:path/to/new/package' │
│ --keep-unlocked Keep the original unlocked q files once the locked │
│ versions are created. │
│ --lock-q-files Lock the contents of the q files in the package. │
│ Files whose first line is /dnc or //dnc will be │
│ ignored. │
│ --all-deps Package all dependencies together. │
│ --tag Tag for release, omit the random dev hash suffix. │
│ --version TEXT Override the version of the package. │
│ --package-name TEXT Override the auto-generated package name. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
.kxi package files are often referred to on CLI and docs as artifacts
To unpack the artifact and view its contents on your local computer, run the kxi package unpack command, as follows:
kxi package unpack --help
Usage: kxi package unpack [OPTIONS] ARTIFACT_PATH
Unpack an artifact to a specified location.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --unpack-dir PATH Path to the directory where the artifact will be │
│ unpacked to;[default] `.`. │
│ --force Force unpack: Suppress user confirmation during │
│ removal of existing package │
│ --clear-target-dir Clear the target directory entirely before │
│ unpacking the artifact. Destructive. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
The kxi unpack command opens the artifact and the kxi packit command creates it again.
The kxi package checkpoint command
The kxi package checkpoint command creates a package artifact given a source code directory. This is useful for packaging applications for deployment within the KX Insights environment.
Run the --help command for more information, as shown below:
kxi package checkpoint --help
Usage: kxi package checkpoint [OPTIONS] SOURCE
Update the package version given a source code directory.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --version TEXT Override the version of the │
│ package. │
│ --bump [major|minor|patch|dev] Increment the │
│ major/minor/patch/dev version. │
│ --version-info TEXT Leave a description note for the │
│ checkpoint version. │
│ --package-name TEXT Override the auto-generated │
│ package name. │
│ --kxi TEXT Create a kxi artifact after │
│ checkpointing in a target │
│ directory. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
Package customizations
This section provides additional information on each of the options available when using kxi package packit.
By default, package naming and versioning are determined by the contents of the manifest file. For many use cases, this is sufficient for generating packages. However, in Continuous Integration/Continuous Deployment (CI/CD) workflows, relying solely on the manifest can make it difficult to align package versions with release tags.
To simplify version management in automated pipelines, you can use the --version option to specify the version directly, as the version to be used when creating a .kxi package file—typically using the tag from your code repository. Similarly, --tag option helps distinguish between a production version and a development version when generating your package.
The following examples illustrate the difference between generating a package with and without the --tag option:
A --tag is simply a version that is "ready for release". An example is a "tagged" version my-pkg-0.4.1.kxi versus its dev counterpart my-pkg-0.4.1.dev1.kxi.
kxi package packit mypkg
mypkg-0.0.2.dev1.kxi
kxi package packit mypkg --tag
mypkg-0.0.1.kxi
Including --tag omits the random dev_hash that is included by default. This is useful within the context of a CI/CD workflow as it enables the same package to be generated within the testing and distribution phases of your workflow without requiring a change to the underlying code.
Using the --tag, --package-name, and --version options results in changes to the manifest file contained within a .kxi package file. Specifically, your manifest file is updated to reflect the name and version given during the packit phase of your workflow.
Lock code
In some cases, locking q code scripts can be a useful technique for protecting sensitive intellectual property (IP). It makes the content of your files unreadable and creates a locked version of functions, hiding their content.
There are two primary ways to lock q files:
Locking requires PyKX
To lock code, you must install PyKX and it must be operating in 'licensed' mode.
Pipeline specs written as code cannot be locked.
However, the analytics they import can be locked. It is considered best practice to separate the code that executes the pipeline from the libraries that contain sensitive intellectual property (IP).
Lock files locally
You can lock q code locally during packaging with the packit tool by running the kxi package packit command, as follows:
kxi package packit pkg --lock-q-files
The following examples show how to lock all code and either keep or remove the original copies of the q files:
kxi package packit mypkg --tag --lock-q-files
kxi package packit mypkg --tag --lock-q-files --keep-unlocked
Standalone lock command for q functions.
This offers the same functionality as packit --lock-q-code but operates directly on the package directory and does not produce a .kxi package file.
kxi package lock --help
Usage: kxi package lock [OPTIONS] SOURCE
Lock the q files within a source directory. Files whose first line is /dnc or
//dnc will be ignored.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --keep-unlocked Keep the original unlocked q files once the locked │
│ versions are created. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
Lock files in memory and on the server
Locking files in memory and on the server is the preferred option, as it doesn't lock file locally and you can still access them.
This method locks q code during the package push process. Use the --lock-q-files with the push command, as follows:
kxi pm push pkg --lock-q-files
Selectively lock q files
Use /dnc or //dnc at the top of q files you want to remain open (e.g., public APIs). By putting /dnc or //dnc at the top of a q file, you can prevent the file from being locked, even when the lock or packit --lock-q-files commands are run.
This is useful for keeping public functions open so they can be extended easily, while locking any private libraries that are used within them.