Skip to content

Packing, installing and locking

Before continuing

Make sure you have read through:

Packing

Packed artifacts live in KX_ARTIFACT_PATH

Generation of a package artifact for upload to your kdb Insights Enterprise instance is facilitated through the use of the command kxi package packit which is defined as follows:

kxi package packit --help
Usage: kxi package packit [OPTIONS] SOURCE

  Create a package artifact given a source code directory.

Options:
  --override-deps TEXT  Specify which deps to override during packit. format
                        'name/ver:path/to/new/package'
  --keep-unlocked       Keep the original unlocked q files once the locked
                        versions are created.
  --lock-q-files        Lock the contents of the q files in the package not.
  --all-deps            Package all dependencies together.
  --tag                 Tag for release, omit the random dev hash suffix.
  --version TEXT        Override the version of the package.
  --package-name TEXT   Override the auto-generated package name.
  --help                Show this message and exit.

This provides you with flexibility in how to bundle packages for upload depending on business requirements. The following provides some additional information on each of the options available allowing you to customize the creation of a package artifact locally or within your Continuous Integration (CI) environment.

The naming and versioning of packages is by default determined by the contents of the manifest file contained within the package. For many use-cases this is sufficient to allow packages to be generated, however, when automating this within Continuous Integration/Continuous Deployment (CI/CD) processes, this limitation can make the process of updating versions to align with release artifacts difficult. The use of --version in particular allows you to take the tag version of the code repository defining your package as the version to be used when creating an artifact. In a similar vein --tag provides the ability to determine, when generating your package, what is a production and what is a development version of your package. The following outlines the difference when generating a package using the --tag option:

kxi package packit mypkg
Refreshing package before packing...
Writing mypkg/manifest.yaml
Creating package from mypkg
Package created: /tmp/artifact_store/mypkg-0.0.1_693D6FC5.kxi
mypkg-0.0.1_693D6FC5.kxi

kxi package packit mypkg --tag
Refreshing package before packing...
Writing mypkg/manifest.yaml
Creating package from mypkg
Package created: /tmp/artifact_store/mypkg-0.0.1.kxi
mypkg-0.0.1.kxi

Including --tag will omit the random dev_hash which is included by default. This is useful within the context of a CI/CD workflow as it allows the same package to be generated within the testing and distribution phases of your workflow without requiring a change to the underlying code.

Usage of the --tag, --package-name and --version options results in changes to the manifest file contained within a package artifact. Specifically, your manifest file is updated to reflect the name and version given during the packit phase of your workflow.

Including all Deps

By default, creating a package artifact will not include any code/config defined by the dependencies.

This means, in order to make use of your dependencies in kdb Insights Enterprise, you can either:

  • Ensure all your dependencies are pushed prior to pushing a package that depends on them
  • Ensure all packages are remotely source-able from your cluster
  • Bundle all packages together and bulk install them

This last option is what the --all-deps flag does. It sources all of the dependencies during artifact creation and includes them all in the *.kxi files.

This functionality can be useful but comes at a cost with respect to the size of your artifacts, which now include all dependent packages.

Override Deps

The Override deps flag is an advanced feature and allows you to "swap-out" dependencies for other dependencies - it is intended to be used in development to test out if a new version of a dependency would be compatible with the package.

Other constraints

It must be run in conjunction with --all-deps or it will have no effect. This is because --all-deps builds the full dependency tree.

It also cannot be used with --tag as we don't want to automatically --tag a release version when doing this

Lastly, the overriding dep must be available locally

kxi package init mypkg --force 
kxi package init pkg-a --force 
kxi package init pkg-b --force 
kxi package init pkg-c --force 
kxi package add --to mypkg dep --name pkg-a --version 0.0.1 
kxi package add --to mypkg dep --name pkg-b --version 0.0.1 
echo "----Setup Finished---"
artifact=$(kxi package packit mypkg --all-deps --override-deps pkg-b/0.0.1:pkg-c/0.0.1 2>&1 | tail -1)
kxi package info $artifact
Creating package at location: mypkg
Writing mypkg/manifest.yaml
Creating package at location: pkg-a
Writing pkg-a/manifest.yaml
Creating package at location: pkg-b
Writing pkg-b/manifest.yaml
Creating package at location: pkg-c
Writing pkg-c/manifest.yaml
Writing mypkg/manifest.yaml
Writing mypkg/manifest.yaml
----Setup Finished---
==PATH==
/tmp/artifact_store/mypkg-0.0.1_F6EC0E03.kxi

==OBJECT==
Artifact

==Manifest==
name: mypkg
version: 0.0.1_F6EC0E03
dependencies:
- name: pkg-a
  version: 0.0.1
  path: pkg-a
  kxi: '{{KX_ARTIFACT_PATH}}/pkg-a-0.0.1.kxi'
- name: pkg-c
  version: 0.0.1
  path: /tmp/tmp1u4xp0ap/pkg-c-0.0.1_2C317F63.kxi
  kxi: '{{KX_ARTIFACT_PATH}}/pkg-c-0.0.1.kxi'
metadata:
  pakxtime: '2024-02-12T18:19:22.286467'
entrypoints:
  default: init.q
system:
  _pakx_version: 1.8.3

In the above example pkg-b was replaced with pkg-c

Locking code

In certain circumstances, such as the dissemination of sensitive IP, it may be useful for you to be able to lock q code scripts. This process removes the ability to read the content of files and creates a locked representation of functions, which removes the ability to display the function content.

Locking requires PyKX

To lock code code, you must have PyKX installed and operating in 'licensed' mode.

The following provides you with two examples of its usage.

  • Locking all code and removing the original copies of the q files
  • Locking all code and keeping the original copies of the q files
kxi package packit mypkg --tag --lock-q-files
kxi package packit mypkg --tag --lock-q-files --keep-unlocked

Locking

There is also a standalone lock function which offers the same functionality as packit --lock-q-code but operates directly on the package directory and does not produce an artifact.

kxi package lock --help
Usage: kxi package lock [OPTIONS] SOURCE

  Lock the q files within a source directory.

Options:
  --keep-unlocked  Keep the original unlocked q files once the locked versions
                   are created.
  --help           Show this message and exit.

Installing

Installing is an essential process that unpacks your .kxi object into a well-defined system path that the packaging system looks for in order to load code and resolve objects.

Installed packages live in KX_PACKAGE_PATH

During this process, dependencies will be resolved (and downloaded is necessary/possible).

kxi package install --help
Usage: kxi package install [OPTIONS] [REQUIREMENT_SPECIFIER]

  Install a package given a requirement or path.

Options:
  --override-deps TEXT   Specify which deps to override during install. format
                         'name/ver:path/to/new/package'
  --simple               Simplify the output to name/version format.
  --add-symlink-to PATH  Create a symlink of package_name/version at the
                         specified directory.
  --force                Force install: overwrite existing packages and
                         dependencies.
  --help                 Show this message and exit.

To install a package and it's deps:

kxi package init --force pkg-a
kxi package init --force pkg-b
kxi package init --force pkg-c
kxi package add --to pkg-a dep --name pkg-b --version 0.0.1
kxi package add --to pkg-a dep --name pkg-c --version 0.0.1
kxi package install pkg-a
Creating package at location: pkg-a
Writing pkg-a/manifest.yaml
Creating package at location: pkg-b
Writing pkg-b/manifest.yaml
Creating package at location: pkg-c
Writing pkg-c/manifest.yaml
Writing pkg-a/manifest.yaml
Writing pkg-a/manifest.yaml
pkg-a-0.0.1-local
    └──pkg-b-0.0.1-local
    └──pkg-c-0.0.1-local
{
    "pkg-b": [
        {
            "version": "0.0.1",
            "_status": "InstallationStatus.SUCCESS"
        }
    ],
    "pkg-c": [
        {
            "version": "0.0.1",
            "_status": "InstallationStatus.SUCCESS"
        }
    ],
    "pkg-a": [
        {
            "version": "0.0.1",
            "_status": "InstallationStatus.SUCCESS"
        }
    ]
}

The resulting installed package will be an open directory structure at the KX_PACKAGE_PATH location in the format: pkg_name/pkg_version.

If the KX_PACKAGE_PATH isn't set, it will be set to a temporary directory (which your OS may clean up unexpectedly).

Removing

To remove installed packages and artifacts:

kxi package uninstall --help
Usage: kxi package uninstall [OPTIONS] [OBJECTS]...

  Uninstall specified locally installed packages and artifacts.

Options:
  --artifacts                     Remove available artifacts rather than
                                  installed packages.
  --obj-type [Package|Artifact|Deployment]
                                  Remove available object from system
  --force                         Force uninstall: suppress user confirmation
                                  during removal.
  --help                          Show this message and exit.

In order to remove all:

kxi package uninstall all --force
{
    "myotherpkg": [
        {
            "_status": "DELETED",
            "version": "0.0.1"
        }
    ],
    "pkg-a": [
        {
            "_status": "DELETED",
            "version": "0.0.1"
        }
    ],
    "pkg-b": [
        {
            "_status": "DELETED",
            "version": "0.0.1"
        }
    ],
    "pkg-c": [
        {
            "_status": "DELETED",
            "version": "0.0.1"
        }
    ]
}

kxi package uninstall all --force --obj-type=Artifact
{
    "mypkg": [
        {
            "_status": "DELETED",
            "version": "0.0.1_693D6FC5"
        },
        {
            "_status": "DELETED",
            "version": "0.0.1"
        },
        {
            "_status": "DELETED",
            "version": "0.0.1_F6EC0E03"
        }
    ]
}

kxi package uninstall all --force --obj-type=Deployment
{}

This doesn't teardown deployments

This will simply delete the deployment metadata tracker object