Skip to content

Creating a package

What will be covered here

After reading this section you will be able to:

  • Create a package using the kdb Insights CLI
  • Add code to the package
  • Convert an asembly to a package

Initializing a package

Use the kdb Insights CLI to initialize packages; in particular, use the following command:

kxi package init --help
Usage: kxi package init [OPTIONS] [PATH_TO_PACKAGE]

  Creates a bare package at the specified target path.

  Note: this will not be saved to your KX_PACKAGE_PATH unless `install` is
  explicitly run.

Options:
  --force         Force initialisation: overwrite existing directory if one
                  already exists.
  --reset         Reset initialisation: overwrite the existing manifest only
  --version TEXT  Version of the package to initialise.
  --help          Show this message and exit.

Packages, artifacts and where to find them

  • KX_PACKAGE_PATH is the location where Package objects live, Package objects are simply directory structures that contain a manifest.
  • KX_ARTIFACT_PATH is the location where Artifact objects live (.kxi archives).
  • Packages can be initialized in any empty directory and can exist within a git repository

Adding Artifacts to the Package path (KX_PACKAGE_PATH) and vice versa is not recommended or supported and can result in errors.

In order to create a new package:

  1. Create a new empty directory and move into this location:

    mkdir my_packages
    cd my_packages
    
  2. Initialize a package named test-pkg:

    kxi package init test-pkg --force
    
    Creating package at location: test-pkg
    Writing test-pkg/manifest.yaml
    

On package names...

Packages are not allowed to have an _ in their name. This is a constraint of the Kubernetes runtime where packages are deployed and run.

Once the package has been initialized you can check out what is inside.

For more information, read below. This provides a breakdown of allowed configuration and customization of your package.

Package contents

You can now add code to the newly created package using a text editor of your choice and the CLI

Reserved Files and Directories

All of the below are populated through the use of the CLI

  • Do not use folders named pipelines, reports, databases, router, deployment_config at the root of your package.
  • Do not use the file names udfs.* and manifest.* at the root of your package.

Converting assemblies to packages

From kdb Insights Enterprise version 1.6.0 onwards we supply a conversion function to help users migrate from assemblies to packages.

This is a good route for migrating to the package workflow if you are starting from an existing assembly.

The convert function can take an assembly file (note the file must be exactly one Assembly and not a list) and create a Package with the same name (in a directory beside the orignal Assembly).

This package can be edited and added to and should be deployable "as is" with Assembly behaviour parity:

kxi package convert --help
Usage: kxi package convert [OPTIONS] FILEPATH

  Depending on what object the filepath points to:

      If file contains assembly: Convert an asm  -> package

      If dir  contains Package : Convert package to -> yaml/json (e.g. --fmt
      json)

Options:
  --fmt [yaml|json]  json or yaml format if converting a package to new fmt
  --help             Show this message and exit.

The example below runs the convert with an existing assembly that includes a database, tables and a pipeline:

kxi package -q convert asm.yaml && kxi package info asm
==PATH==
/Users/kx/packages/asm

==OBJECT==
Package

==Manifest==
name: asm
version: 0.0.1
databases:
  asm-db:
    dir: databases/asm-db
    shards:
    - asm-db-shard
    tables:
    - trade
    - quote
    - exchange
    - instrument
pipelines:
  sp-asm:
    file: pipelines/sp-asm.yaml
router: router/router.yaml
deployment_config: deployment_config/deployment_config.yaml

If the conversion fails, it will tell you which fields are problematic

In order to ensure a successful conversion, modify the existing assembly by fixing the required fields. For example, if this is returned:

spec -> elements -> sm -> tiers -> 1 -> compression -> level field required (type=value_error.missing)
spec -> elements -> sm -> tiers -> 2 -> compression -> level field required (type=value_error.missing)
spec -> elements -> sm -> stream
extra fields not permitted (type=value_error.extra)

It shows that the conversion requires the field compression to be added to the 2 tiers and the field "stream" to be removed from the Storage Manager.

Known issues with converting to packages from the UI export

Packages exported from the UI require manual editing, particularly around the exported pipeline format which may require you to remove the persistence and k8sPolicy information, and then re-add them after conversion.

Package structure

A package in its simplest form generated by the CLI consists of the following structure:

kxi package init test-package --force && tree test-package
Creating package at location: test-package
Writing test-package/manifest.yaml
test-package
├── init.q
└── manifest.yaml

0 directories, 2 files

You can extend this to contain any arbitrary code (e.g below), as long as none of the previously mentioned constraints are violated:

└── ml
    ├── init.q
    ├── manifest.yaml
    ├── ml.q
    └── machine_learning
        ├── preprocessing
             └── preproc.q
        ├── model.py
        └── model.q

Using the import API you are able to load the code in q or python.

Manifest File

Manifest format

The manifest file is by default in yaml but can be migrated to json using:

kxi package convert mypkg --fmt json

The manifest.yaml file which is present when a new package is initialized is centrally important to the use of a package. Without a defined manifest.yaml file, a package cannot be used by kdb Insights Enterprise or the APIs provided for package interaction.

On initialization of a package you are presented with a manifest.yaml file with the following structure:

kxi package -q init test-package --force && cat test-package/manifest.yaml
name: test-package
version: 0.0.1
metadata:
  description: ''
  authors:
  - name: root
    email: ''
entrypoints:
  default: init.q

The following table provides a brief description of each configurable section within the manifest.yaml and whether definition of its content is required for the package to be used effectively.

The keys defined at initialization must not be deleted

section                       description required
name The name associated with the package by default when building it. yes
version The version associated with the package by default when building it. yes
entrypoints The set of possible methods by which a package can be loaded. More information available here. no
license The relative path to the license file under which the defined package is intended to be released. no
metadata Information about the package contents and the users who have contributed to it. no
dependencies Any explicit dependencies on additional packages. More information available here. no
system Information about the system conditions under which the package was generated. Presently this includes the version of the pakx cli which was used to create the package. no
pipelines This denotes any Stream Processor pipeline definitions within a package. no
databases This denotes the databases that have been defined within a package. no
udf_namespaces This denotes the tagged names which are searched when parsing the package text for user-defined functions. no

Entrypoints

Entrypoints define the q/Python files which can be used as the initialization script for a package. The default entrypoint used when loading a package is default and is defined as init.q, this file is used when a package is loaded with no specific entrypoint defined. You can update this entrypoint to be any file relative to the package root i.e.

entrypoints:
  default: src/init.q

For more advanced usage you can specify multiple entrypoints for a package, allowing sub-sections of a code-base to be loaded independently. This is particularly useful when attempting to split code based on the area of an application it is intended to be used within, for example the following could define entrypoints specific to the pipelines and data access processes and aggregators separately.

entrypoints:
  default: init.q
  sp: src/sp.q
  data-access: src/da.q
  aggregator: src/agg.q

The API which provides you with the ability to load specific entrypoints is defined in q here and in Python here.

The use of Python entrypoints is currently a beta feature and still in active development.

It is supported only when using the Python API independently of kdb Insights Enterprise; for example, when developing packages for use within kdb Insights Enterprise, entrypoints must at present be defined with a *.q extension.

Package dependencies

The dependencies section of the manifest.yaml file outlines any external dependencies on which the package being defined is explicitly dependent.

Installation

The package can installed by running the following:

kxi package install mypkg

If any dependencies are specified in the manifest, it will resolve each of them and install them on the host system (i.e. the system where the command has been run).

Dependency format

The expected structure for defining dependencies is as follows:

dependencies:
    - location: loc
      repo: repo
      version: ver

The keys within this dependency structure relate to the following:

key description
name The name of the package to be retrieved as a dependency
version The version of the package dependency-name which is to be retrieved as a dependency
location The storage location from which a package is to be retrieved, one of local, github, gitlab or kx-nexus
repo The repository URL from which the dependency is to be retrieved
path The local path from which the dependency is to be retrieved
kxi A special field pointing to the artifact location that is used when nested dependencies are used

For completeness each location option is outlined below separately and the underlying structure of the request completed when retrieving the requested dependency is shown.

Required environment variables:

  • GITHUB_TOKEN this token is required to allow you to download artifacts from github and can be generated by following the instructions outlined here

kxi package -q init pkg-with-dep --force
kxi package -q add --to pkg-with-dep dep --location github --repo test_user/test_repo --name test-package --version 1.0.0
cat pkg-with-dep/manifest.yaml | grep "dependencies" -A 5
dependencies:
- name: test-package
  version: 1.0.0
  repo: test_user/test_repo
  location: github
metadata:

The underlying URL against which this request is executed is as following (this can be seen in the pakx-config):

https://github.com/{package.repo}/release/download/{package.version}/{package.name}-{package.version}.kxi

Required environment variables:

  • GITLAB_TOKEN this token is required to allow you to download artifacts from gitlab and can be generated by following the instructions outline here

kxi package -q init pkg-with-dep --force
kxi package -q add --to pkg-with-dep dep --location gitlab --repo test_user/test_repo --name test-package --version 1.0.0
cat pkg-with-dep/manifest.yaml | grep "dependencies" -A 5
dependencies:
- name: test-package
  version: 1.0.0
  repo: test_user/test_repo
  location: gitlab
metadata:

The underlying URL against which this request is executed is as following (this can be seen in the pakx-config):

https://gitlab.com/api/v4/projects/{package.repo}/packages/generic/{package.name}/{package.version}/{package.name}-{package.version}.kxi

Required environment variables:

  • KX_NEXUS_USER/KX_NEXUS_PASS The username/password associated with your access to the KX External Nexus

kxi package -q init pkg-with-dep --force 
kxi package -q add --to pkg-with-dep dep --location kx-nexus --repo test_user/test_repo --name test-package --version 1.0.0
cat pkg-with-dep/manifest.yaml | grep "dependencies" -A 5
dependencies:
- name: test-package
  version: 1.0.0
  repo: test_user/test_repo
  location: kx-nexus
metadata:

The underlying URL against which this request is executed is as following (this can be seen in the pakx-config):

https://nexus.kxi-dev.kx.com/repository/{package.repo}/{package.name}/{package.version}/{package.name}-{package.version}.kxi"

Configuration of additional dependencies options can be done within the ~/.insights/pakx-config config which allows for modifications to the retrieval locations or added sources.

Local dependencies

When adding local package dependencies:

  • path should be the absolute filepath
  • Specifying name and version and location: local is allowed
  • Searching path ordering is generally . followed by KXI_PACKAGE_PATH and finally KXI_ARTIFACT_PATH

The following is an example request which would find test-package.1.0.0.kxi based on a tagged release 1.0.0 stored at the location path/to/ on the local host.

Deps have location: local by default

deps:
    - path: path/to/test-package-1.0.0.kxi
export KX_ARTIFACT_PATH=/path/to
deps:
    - name: test-package
      version: 0.0.1

The version in the yaml will take precedence over the version in the filepath

Next steps