Packing Installing and locking
Packing
Packed Artifacts live in KX_ARTIFACT_PATH
Generation of a package artifact for upload to your kdb Insights Enterprise instance is facilitated through the use of the command kxi package packit which is defined as follows:
kxi package packit --help
Usage: kxi package packit [OPTIONS] SOURCE
Create a package artifact given a source code directory.
Options:
--override-deps TEXT Specify which deps to override during packit. format
'name/ver:path/to/new/package'
--keep-unlocked Keep the original unlocked q files once the locked
versions are created.
--lock-q-files Lock the contents of the q files in the package not.
--all-deps Package all dependencies together.
--tag Tag for release, omit the random dev hash suffix.
--version TEXT Override the version of the package.
--package-name TEXT Override the auto-generated package name.
--help Show this message and exit.
This provides you with flexibility in how to bundle packages for upload depending on business requirements. The following provides some additional information on each of the options available allowing you to customize the creation of a package artifact locally or within your Continuous Integration (CI) environment.
The naming and versioning of packages is by default determined by the contents of the manifest file contained within the package. For many use-cases this is sufficient to allow packages to be generated, however, when automating this within Continuous Integration/Continuous Deployment (CI/CD) processes, this limitation can make the process of updating versions to align with release artifacts difficult. The use of --version in particular allows you to take the tag version of the code repository defining your package as the version to be used when creating an artifact. In a similar vein --tag provides the ability to determine, when generating your package, what is a production and what is a development version of your package. The following outlines the difference when generating a package using the --tag option:
kxi package packit mypkg
Refreshing package before packing...
Writing mypkg/manifest.yaml
Creating package from mypkg
Package created: /tmp/artifact_store/mypkg-0.0.1_26DA6779.kxi
mypkg-0.0.1_26DA6779.kxi
kxi package packit mypkg --tag
Refreshing package before packing...
Writing mypkg/manifest.yaml
Creating package from mypkg
Package created: /tmp/artifact_store/mypkg-0.0.1.kxi
mypkg-0.0.1.kxi
Including --tag will omit the random dev_hash which is included by default. This is useful within the context of a CI/CD workflow as it allows the same package to be generated within the testing and distribution phases of your workflow without requiring a change to the underlying code.
Usage of the --tag, --package-name and --version options results in changes to the manifest file contained within a package artifact. Specifically, your manifest file is updated to reflect the name and version given during the packit phase of your workflow.
Including All Deps
By default, creating a package artifact will not include any code/config defined by the dependencies.
This means, in order to make use of dependencies your dependencies in kdb Insights Enterprise, you can either:
- Ensure all your dependencies are pushed prior to pushing a package that depends on them
- Ensure all packages are remotely source-able from your cluster
- Bundle all packages together and bulk install them
This last option is what the --all-deps flag does. It sources all of the dependencies during artifact creation and includes them all in the *.kxi files.
This functionality can be useful but comes at a cost with respect to the size of your artifacts, which now include all dependent packages.
Override Deps
The Override deps flag is an advanced feature and allows the user to "swap-out" dependencies for other dependencies - it is intended to be used in development to test out if a new version of a dependency would be compatible with the package.
Other constraints
It must be run in conjunction with --all-deps or it will have no effect. This is because --all-deps builds the full dependency tree.
It also cannot be used with --tag as we don't want to automatically --tag a release version when doing this
Lastly, the overriding dep must be available locally
kxi package init mypkg --force
kxi package init pkg-a --force
kxi package init pkg-b --force
kxi package init pkg-c --force
kxi package add --to mypkg dep --name pkg-a --version 0.0.1
kxi package add --to mypkg dep --name pkg-b --version 0.0.1
echo "----Setup Finished---"
artifact=$(kxi package packit mypkg --all-deps --override-deps pkg-b/0.0.1:pkg-c/0.0.1 2>&1 | tail -1)
kxi package info $artifact
Creating package at location: mypkg
Writing mypkg/manifest.yaml
Creating package at location: pkg-a
Writing pkg-a/manifest.yaml
Creating package at location: pkg-b
Writing pkg-b/manifest.yaml
Creating package at location: pkg-c
Writing pkg-c/manifest.yaml
Writing mypkg/manifest.yaml
Writing mypkg/manifest.yaml
----Setup Finished---
==PATH==
/tmp/artifact_store/mypkg-0.0.1_1FE6401A.kxi
==OBJECT==
Artifact
==Manifest==
name: mypkg
version: 0.0.1_1FE6401A
dependencies:
- name: pkg-a
version: 0.0.1
path: pkg-a
kxi: '{{KX_ARTIFACT_PATH}}/pkg-a-0.0.1.kxi'
- name: pkg-c
version: 0.0.1
path: /tmp/tmpz94hlx65/pkg-c-0.0.1_4C9B236.kxi
kxi: '{{KX_ARTIFACT_PATH}}/pkg-c-0.0.1.kxi'
metadata:
pakxtime: '2024-04-19T06:09:04.403027'
entrypoints:
default: init.q
system:
_pakx_version: 1.8.3
In the above example we replaced pkg-b with pkg-c
Locking Code
In certain circumstances, such as the dissemination of sensitive IP, it may be useful for you to be able to lock q code scripts. This process removes the ability to read the content of files and creates a locked representation of functions, which removes the ability to display the function content.
Locking requires PyKX
To lock code code, you must have PyKX installed and operating in 'licensed' mode.
The following provides you with two examples of its usage.
- Locking all code and removing the original copies of the q files
- Locking all code and keeping the original copies of the q files
kxi package packit mypkg --tag --lock-q-files
kxi package packit mypkg --tag --lock-q-files --keep-unlocked
Locking
There is also a standalone lock function which offers the same functionality as packit --lock-q-code but operates directly on the package directory and does not produce an artifact.
kxi package lock --help
Usage: kxi package lock [OPTIONS] SOURCE
Lock the q files within a source directory.
Options:
--keep-unlocked Keep the original unlocked q files once the locked versions
are created.
--help Show this message and exit.
Installing
Installing is an essential process that unpacks your .kxi object into a well-defined system path that the packaging system looks for in order to load code and resolve objects.
Installed packages live in KX_PACKAGE_PATH
During this process, dependencies will be resolved (and downloaded is necessary/possible).
kxi package install --help
Usage: kxi package install [OPTIONS] [REQUIREMENT_SPECIFIER]
Install a package given a requirement or path.
Options:
--override-deps TEXT Specify which deps to override during install. format
'name/ver:path/to/new/package'
--simple Simplify the output to name/version format.
--add-symlink-to PATH Create a symlink of package_name/version at the
specified directory.
--force Force install: overwrite existing packages and
dependencies.
--help Show this message and exit.
To install a package and it's deps:
kxi package init --force pkg-a
kxi package init --force pkg-b
kxi package init --force pkg-c
kxi package add --to pkg-a dep --name pkg-b --version 0.0.1
kxi package add --to pkg-a dep --name pkg-c --version 0.0.1
kxi package install pkg-a
Creating package at location: pkg-a
Writing pkg-a/manifest.yaml
Creating package at location: pkg-b
Writing pkg-b/manifest.yaml
Creating package at location: pkg-c
Writing pkg-c/manifest.yaml
Writing pkg-a/manifest.yaml
Writing pkg-a/manifest.yaml
pkg-a-0.0.1-local
└──pkg-b-0.0.1-local
└──pkg-c-0.0.1-local
{
"pkg-b": [
{
"version": "0.0.1",
"_status": "InstallationStatus.SUCCESS"
}
],
"pkg-c": [
{
"version": "0.0.1",
"_status": "InstallationStatus.SUCCESS"
}
],
"pkg-a": [
{
"version": "0.0.1",
"_status": "InstallationStatus.SUCCESS"
}
]
}
KX_PACKAGE_PATH location in the format: pkg_name/pkg_version.
If the KX_PACKAGE_PATH isn't set, it will be set to a temporary directory (which your OS may clean up unexpectedly).
Removing
To remove installed packages and artifacts:
kxi package uninstall --help
Usage: kxi package uninstall [OPTIONS] [OBJECTS]...
Uninstall specified locally installed packages and artifacts.
Options:
--artifacts Remove available artifacts rather than
installed packages.
--obj-type [Package|Artifact|Deployment]
Remove available object from system
--force Force uninstall: suppress user confirmation
during removal.
--help Show this message and exit.
In order to remove all:
kxi package uninstall all --force
{
"myotherpkg": [
{
"_status": "DELETED",
"version": "0.0.1"
}
],
"pkg-a": [
{
"_status": "DELETED",
"version": "0.0.1"
}
],
"pkg-b": [
{
"_status": "DELETED",
"version": "0.0.1"
}
],
"pkg-c": [
{
"_status": "DELETED",
"version": "0.0.1"
}
]
}
kxi package uninstall all --force --obj-type=Artifact
{
"mypkg": [
{
"_status": "DELETED",
"version": "0.0.1_26DA6779"
},
{
"_status": "DELETED",
"version": "0.0.1"
},
{
"_status": "DELETED",
"version": "0.0.1_1FE6401A"
}
]
}
kxi package uninstall all --force --obj-type=Deployment
{}
This doesn't teardown deployments
This will simply delete the deployment metadata tracker object