Skip to content

Pack a Package

This page describes how to pack a package to a single sharable .kxi in kdb Insights Enterprise.

Before uploading a package to a kdb Insights Enterprise instance, you have the option to create a kxi package file out of the package, which is stored in the KX_ARTIFACT_PATH. This may be useful when sharing a package with another user or for other distribution purposes.

This is facilitated using the command kxi package packit, which allows you to customize the creation of a .kxi package file either locally or within your Continuous Integration (CI) environment. This command is defined as follows:

kxi package packit --help
 Usage: kxi package packit [OPTIONS] SOURCE                                     

 Create a package artifact given a source code directory.                       

╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --override-deps    TEXT  Specify which deps to override during packit.       │
│                          format 'name/ver:path/to/new/package'               │
│ --keep-unlocked          Keep the original unlocked q files once the locked  │
│                          versions are created.                               │
│ --lock-q-files           Lock the contents of the q files in the package.    │
│                          Files whose first line is /dnc or //dnc will be     │
│                          ignored.                                            │
│ --all-deps               Package all dependencies together.                  │
│ --tag                    Tag for release, omit the random dev hash suffix.   │
│ --version          TEXT  Override the version of the package.                │
│ --package-name     TEXT  Override the auto-generated package name.           │
│ --help                   Show this message and exit.                         │
╰──────────────────────────────────────────────────────────────────────────────╯

.kxi package files are often referred to on CLI and docs as artifacts

To unpack the artifact and view its contents on your local computer, run the kxi package unpack command, as follows:

kxi package unpack --help
 Usage: kxi package unpack [OPTIONS] ARTIFACT_PATH                              

 Unpack an artifact to a specified location.                                    

╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --unpack-dir          PATH  Path to the directory where the artifact will be │
│                             unpacked to;[default] `.`.                       │
│ --force                     Force unpack: Suppress user confirmation during  │
│                             removal of existing package                      │
│ --clear-target-dir          Clear the target directory entirely before       │
│                             unpacking the artifact. Destructive.             │
│ --help                      Show this message and exit.                      │
╰──────────────────────────────────────────────────────────────────────────────╯

The kxi unpack command opens the artifact and the kxi packit command creates it again.

The kxi package checkpoint command

The kxi package checkpoint command creates a package artifact given a source code directory. This is useful for packaging applications for deployment within the KX Insights environment.

Run the --help command for more information, as shown below:

kxi package checkpoint --help
 Usage: kxi package checkpoint [OPTIONS] SOURCE                                 

 Create a package artifact given a source code directory.                       

╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --version         TEXT                     Override the version of the       │
│                                            package.                          │
│ --bump            [major|minor|patch|dev]  Increment the                     │
│                                            major/minor/patch/dev version.    │
│ --version-info    TEXT                     Leave a description note for the  │
│                                            checkpoint version.               │
│ --package-name    TEXT                     Override the auto-generated       │
│                                            package name.                     │
│ --kxi             TEXT                     Create a kxi artifact after       │
│                                            checkpointing in a target         │
│                                            directory.                        │
│ --help                                     Show this message and exit.       │
╰──────────────────────────────────────────────────────────────────────────────╯

Package customizations

This section provides additional information on each of the options available when using kxi package packit.

By default, the naming and versioning of packages are determined by the contents of the manifest file. For many use cases, this is sufficient for generating packages. However, when automating this in Continuous Integration/Continuous Deployment (CI/CD) processes, this limitation can complicate the process of updating versions to match release .kxi package files.

To address this, you can use the --version option to take the tag version of the code repository defining your package as the version to be used when creating a .kxi package file. Similarly, --tag helps distinguish between a production version and a development version when generating your package.

The following examples illustrate the difference between generating a package with and without the --tag option:

A --tag is simply a version that is "ready for release". An example is a "tagged" version my-pkg-0.4.1.kxi versus its dev counterpart my-pkg-0.4.1.dev1.kxi.

kxi package packit mypkg
mypkg-0.0.2.dev1.kxi

kxi package packit mypkg --tag
mypkg-0.0.1.kxi

Including --tag omits the random dev_hash that is included by default. This is useful within the context of a CI/CD workflow as it enables the same package to be generated within the testing and distribution phases of your workflow without requiring a change to the underlying code.

Using the --tag, --package-name, and --version options results in changes to the manifest file contained within a .kxi package file. Specifically, your manifest file is updated to reflect the name and version given during the packit phase of your workflow.

Lock code

In some situations, like when sharing sensitive intellectual property (IP), it can be helpful to lock q code scripts. This process prevents the content of the files from being read and creates a locked version of functions, which hides their content.

Standalone lock command for q functions.

This offers the same functionality as packit --lock-q-code but operates directly on the package directory and does not produce a .kxi package file.

kxi package lock --help
 Usage: kxi package lock [OPTIONS] SOURCE                                       

 Lock the q files within a source directory. Files whose first line is /dnc or  
 //dnc will be ignored.                                                         

╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --keep-unlocked      Keep the original unlocked q files once the locked      │
│                      versions are created.                                   │
│ --help               Show this message and exit.                             │
╰──────────────────────────────────────────────────────────────────────────────╯

Locking requires PyKX

To lock code, you must install PyKX and it must be operating in 'licensed' mode.

The following examples show how to lock all code and either keep or remove the original copies of the q files:

kxi package packit mypkg --tag --lock-q-files
kxi package packit mypkg --tag --lock-q-files --keep-unlocked

Pipeline specs written as code cannot be locked.

However, the analytics they import can be locked. It is considered best practice to separate the code that executes the pipeline from the libraries that contain sensitive intellectual property (IP).

Selectively locking q files

By putting /dnc or //dnc at the top of a q file, you can prevent the file from being locked, even when the lock or packit --lock-q-files commands are run. This is useful for keeping public functions open so they can be extended easily, while locking any private libraries that are used within them.

Next steps