Pack a Package
This page describes how to pack a package to a single sharable .kxi
in kdb Insights Enterprise.
Before uploading a package to a kdb Insights Enterprise instance, you have the option to create a kxi
package file out of the package, which is stored in the KX_ARTIFACT_PATH
. This may be useful when sharing a package with another user or for other distribution purposes.
This is facilitated using the command kxi package packit
, which allows you to customize the creation of a .kxi
package file either locally or within your Continuous Integration (CI) environment. This command is defined as follows:
kxi package packit --help
Usage: kxi package packit [OPTIONS] SOURCE
Create a package artifact given a source code directory.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --override-deps TEXT Specify which deps to override during packit. │
│ format 'name/ver:path/to/new/package' │
│ --keep-unlocked Keep the original unlocked q files once the locked │
│ versions are created. │
│ --lock-q-files Lock the contents of the q files in the package. │
│ Files whose first line is /dnc or //dnc will be │
│ ignored. │
│ --all-deps Package all dependencies together. │
│ --tag Tag for release, omit the random dev hash suffix. │
│ --version TEXT Override the version of the package. │
│ --package-name TEXT Override the auto-generated package name. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
.kxi
package files are often referred to on CLI and docs as artifacts
To unpack
the artifact and view its contents on your local computer, run the kxi package unpack
command, as follows:
kxi package unpack --help
Usage: kxi package unpack [OPTIONS] ARTIFACT_PATH
Unpack an artifact to a specified location.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --unpack-dir PATH Path to the directory where the artifact will be │
│ unpacked to;[default] `.`. │
│ --force Force unpack: Suppress user confirmation during │
│ removal of existing package │
│ --clear-target-dir Clear the target directory entirely before │
│ unpacking the artifact. Destructive. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
The kxi unpack
command opens the artifact and the kxi packit
command creates it again.
The kxi package checkpoint
command
The kxi package checkpoint
command creates a package artifact given a source code directory. This is useful for packaging applications for deployment within the KX Insights environment.
Run the --help command for more information, as shown below:
kxi package checkpoint --help
Usage: kxi package checkpoint [OPTIONS] SOURCE
Create a package artifact given a source code directory.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --version TEXT Override the version of the │
│ package. │
│ --bump [major|minor|patch|dev] Increment the │
│ major/minor/patch/dev version. │
│ --version-info TEXT Leave a description note for the │
│ checkpoint version. │
│ --package-name TEXT Override the auto-generated │
│ package name. │
│ --kxi TEXT Create a kxi artifact after │
│ checkpointing in a target │
│ directory. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
Package customizations
This section provides additional information on each of the options available when using kxi package packit
.
By default, the naming and versioning of packages are determined by the contents of the manifest
file. For many use cases, this is sufficient for generating packages. However, when automating this in Continuous Integration/Continuous Deployment (CI/CD) processes, this limitation can complicate the process of updating versions to match release .kxi
package files.
To address this, you can use the --version
option to take the tag version of the code repository defining your package as the version to be used when creating a .kxi
package file. Similarly, --tag
helps distinguish between a production
version and a development
version when generating your package.
The following examples illustrate the difference between generating a package with and without the --tag
option:
A --tag
is simply a version that is "ready for release". An example is a "tagged" version my-pkg-0.4.1.kxi
versus its dev counterpart my-pkg-0.4.1.dev1.kxi
.
kxi package packit mypkg
mypkg-0.0.2.dev1.kxi
kxi package packit mypkg --tag
mypkg-0.0.1.kxi
Including --tag
omits the random dev_hash
that is included by default. This is useful within the context of a CI/CD workflow as it enables the same package to be generated within the testing and distribution phases of your workflow without requiring a change to the underlying code.
Using the --tag
, --package-name
, and --version
options results in changes to the manifest
file contained within a .kxi
package file. Specifically, your manifest
file is updated to reflect the name and version given during the packit
phase of your workflow.
Lock code
In some situations, like when sharing sensitive intellectual property (IP), it can be helpful to lock q code scripts. This process prevents the content of the files from being read and creates a locked
version of functions, which hides their content.
Standalone lock
command for q functions.
This offers the same functionality as packit --lock-q-code
but operates directly on the package directory and does not produce a .kxi
package file.
kxi package lock --help
Usage: kxi package lock [OPTIONS] SOURCE
Lock the q files within a source directory. Files whose first line is /dnc or
//dnc will be ignored.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --keep-unlocked Keep the original unlocked q files once the locked │
│ versions are created. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
Locking requires PyKX
To lock code, you must install PyKX
and it must be operating in 'licensed' mode.
The following examples show how to lock all code and either keep or remove the original copies of the q files:
kxi package packit mypkg --tag --lock-q-files
kxi package packit mypkg --tag --lock-q-files --keep-unlocked
Pipeline specs written as code cannot be locked.
However, the analytics they import can be locked. It is considered best practice to separate the code that executes the pipeline from the libraries that contain sensitive intellectual property (IP).
Selectively locking q files
By putting /dnc
or //dnc
at the top of a q file, you can prevent the file from being locked, even when the lock
or packit --lock-q-files
commands are run.
This is useful for keeping public functions open so they can be extended easily, while locking any private libraries that are used within them.