Creating a package

Before Continuing...

Make sure that you've:

What will be covered here

After reading this section you will be able to:

Create a package using the kdb Insights CLI
Add code to this package
Convert an Assembly to a Package

Initializing a package

Use the kdb Insights CLI to initialize packages; in particular, use the following command:

kxi package init --help

Usage: kxi package init [OPTIONS] [PATH_TO_PACKAGE]

  Creates a bare package at the specified target path.

  Note: this will not be saved to your KX_PACKAGE_PATH unless `install` is
  explicitly run.

Options:
  --force         Force initialisation: overwrite existing directory if one
                  already exists.
  --reset         Reset initialisation: overwrite the existing manifest only
  --version TEXT  Version of the package to initialise.
  --help          Show this message and exit.

Packages, Artifacts and where to find them

KX_PACKAGE_PATH is the location where Package objects live, Package objects are simply directory structures that contain a manifest.
KX_ARTIFACT_PATH is the location where Artifact objects live (.kxi archives).
Adding Artifacts to the Package path (KX_PACKAGE_PATH) and vice versa is not recommended or supported and can result in errors!
Packages can be initialized in any empty directory and can exist within a git repository

In order to create a new package:

Create a new empty directory and move into this location
```
mkdir my_packages
cd my_packages
```

Initialize a package named test-pkg

kxi package init test-pkg --force

Creating package at location: test-pkg
Writing test-pkg/manifest.yaml

On Package Names...

Packages aren't allowed to have _ in their name. This is a constraint of the Kubernetes runtime where we can deploy and run packages.

Once the package has been initialized you can check out what's inside.

For more information, read below. This provides a breakdown of allowed configuration and customization of your package.

Package contents

You can now add code to the newly created package using a text editor of your choice and the CLI

Reserved Files And Directories

Reserved Files and Directories

All of the below are populated through the use of the CLI

Do not use folders named pipelines, reports, databases, router, deployment_config at the root of your package.
Do not use the file names udfs.* and manifest.* at the root of your package.

Converting Assemblies to Packages

From version 1.6.0 onwards we supply a conversion function to help users migrate from assemblies to packages.

This is a good route for migrating to the package workflow if you're starting from an existing assembly.

The convert function can take an assembly file (note the file must be exactly one Assembly and not a list) and create a Package with the same name (in a directory beside the orignal Assembly).

This package can be edited and added to and should be deployable "as is" with Assembly behaviour parity.

pakx convert --help

Usage: pakx convert [OPTIONS] FILEPATH

  Depending on what object the filepath points to:

      If file contains assembly: Convert an asm  -> package

      If dir  contains Package : Convert package to -> yaml/json (e.g. --fmt
      json)

Options:
  --fmt [yaml|json]  json or yaml format if converting a package to new fmt
  --help             Show this message and exit.

Testing it out below with an existing assembly with a database, tables and a pipeline.

kxi package -q convert asm.yaml && kxi package info asm

==PATH==
/Users/kx/packages/asm

==OBJECT==
Package

==Manifest==
name: asm
version: 0.0.1
databases:
  asm-db:
    dir: databases/asm-db
    shards:
    - asm-db-shard
    tables:
    - trade
    - quote
    - exchange
    - instrument
pipelines:
  sp-asm:
    file: pipelines/sp-asm.yaml
router: router/router.yaml
deployment_config: deployment_config/deployment_config.yaml

Package Structure

A package in its simplest form generated by the CLI consists of the following structure:

kxi package init test-package --force && tree test-package

Creating package at location: test-package
Writing test-package/manifest.yaml
test-package
├── init.q
└── manifest.yaml

0 directories, 2 files

You can extend this to contain any arbitrary code (e.g below), as long as none of the previously mentioned constraints are violated:

└── ml
    ├── init.q
    ├── manifest.yaml
    ├── ml.q
    └── machine_learning
        ├── preprocessing
        │     └── preproc.q
        ├── model.py
        └── model.q

Using the import API you are able to load the code in q or python.

Manifest File

Manifest format

The manifest file is by default in yaml but can be migrated to json using:

kxi package convert mypkg --fmt json

The manifest.yaml file which is present when a new package is initialized is centrally important to the use of a package. Without a defined manifest.yaml file, a package cannot be used by kdb Insights Enterprise or the APIs provided for package interaction.

On initialization of a package you are presented with a manifest.yaml file with the following structure:

kxi package -q init test-package --force && cat test-package/manifest.yaml

name: test-package
version: 0.0.1
metadata:
  description: ''
  authors:
  - name: root
    email: ''
entrypoints:
  default: init.q

The following table provides a brief description of each configurable section within the manifest.yaml and whether definition of its content is required for the package to be used effectively.

Warning

The keys defined at initialization must not be deleted.

section	description	required
`name`	The name associated with the package by default when building it.	`yes`
`version`	The version associated with the package by default when building it.	`yes`
`entrypoints`	The set of possible methods by which a package can be loaded. More information available here.	`no`
`license`	The relative path to the license file under which the defined package is intended to be released.	`no`
`metadata`	Information about the package contents and the users who have contributed to it.	`no`
`dependencies`	Any explicit dependencies on additional packages. More information available here.	`no`
`system`	Information about the system conditions under which the package was generated. Presently this includes the version of the `pakx` cli which was used to create the package.	`no`
`pipelines`	This denotes any Stream Processor pipeline definitions within a package.	`no`
`databases`	This denotes the databases that have been defined within a package.	`no`
`udf_namespaces`	This denotes the tagged names which are searched when parsing the package text for user-defined functions.	`no`

Entrypoints

Entrypoints define the q/Python files which can be used as the initialization script for a package. The default entrypoint used when loading a package is default and is defined as init.q, this file is used when a package is loaded with no specific entrypoint defined. You can update this entrypoint to be any file relative to the package root i.e.

entrypoints:
  default: src/init.q

For more advanced usage you can specify multiple entrypoints for a package, allowing sub-sections of a code-base to be loaded independently. This is particularly useful when attempting to split code based on the area of an application it is intended to be used within, for example the following could define entrypoints specific to the pipelines and data access processes and aggregators separately.

entrypoints:
  default: init.q
  sp: src/sp.q
  data-access: src/da.q
  aggregator: src/agg.q

The API which provides you with the ability to load specific entrypoints is defined in q here and in Python here.

Warning

The use of Python entrypoints is currently a beta feature and still in active development. It is supported only when using the Python API independently of kdb Insights Enterprise; for example, when developing packages for use within kdb Insights Enterprise, entrypoints must at present be defined with a *.q extension.

Package Dependencies

The dependencies section of the manifest.yaml file outlines any external dependencies on which the package being defined is explicitly dependent.

Installation

When we run

kxi package install mypkg

If any dependencies are specified in the manifest, it will resolve each of them and install them on the host system (i.e. the system where the command has been run).

Dependency format

The expected structure for defining dependencies is as follows:

dependencies:
    - location: loc
      repo: repo
      version: ver

The keys within this dependency structure relate to the following:

key	description
`name`	The name of the package to be retrieved as a dependency.
`version`	The version of the package `dependency-name` which is to be retrieved as a dependency.
`location`	The storage location from which a package is to be retrieved, one of `local`, `github`, `gitlab` or `kx-nexus`.
`repo`	The repository URL from which the dependency is to be retrieved.
`path`	The local path from which the dependency is to be retrieved.
`kxi`	A special field pointing to the artifact location that is used when nested dependencies are used.

For completeness we will outline each location option separately and the underlying structure of the request completed when retrieving the requested dependency.

GithubGitlabkx-nexus

Required environment variables:

GITHUB_TOKEN this token is required to allow a user download artifacts from github and can be generated by following the instructions outlined here

kxi package -q init pkg-with-dep --force
kxi package -q add --to pkg-with-dep dep --location github --repo test_user/test_repo --name test-package --version 1.0.0
cat pkg-with-dep/manifest.yaml | grep "dependencies" -A 5

dependencies:
- name: test-package
  version: 1.0.0
  repo: test_user/test_repo
  location: github
metadata:

The underlying URL against which this request is executed is as following (this can be seen in the pakx-config):

https://github.com/{package.repo}/release/download/{package.version}/{package.name}-{package.version}.kxi

Required environment variables:

GITLAB_TOKEN this token is required to allow a user download artifacts from gitlab and can be generated by following the instructions outline here

kxi package -q init pkg-with-dep --force
kxi package -q add --to pkg-with-dep dep --location gitlab --repo test_user/test_repo --name test-package --version 1.0.0
cat pkg-with-dep/manifest.yaml | grep "dependencies" -A 5

dependencies:
- name: test-package
  version: 1.0.0
  repo: test_user/test_repo
  location: gitlab
metadata:

The underlying URL against which this request is executed is as following (this can be seen in the pakx-config):

https://gitlab.com/api/v4/projects/{package.repo}/packages/generic/{package.name}/{package.version}/{package.name}-{package.version}.kxi

Required environment variables:

KX_NEXUS_USER/KX_NEXUS_PASS The username/pass associated with a users access to the KX External Nexus

kxi package -q init pkg-with-dep --force 
kxi package -q add --to pkg-with-dep dep --location kx-nexus --repo test_user/test_repo --name test-package --version 1.0.0
cat pkg-with-dep/manifest.yaml | grep "dependencies" -A 5

dependencies:
- name: test-package
  version: 1.0.0
  repo: test_user/test_repo
  location: kx-nexus
metadata:

The underlying URL against which this request is executed is as following (this can be seen in the pakx-config):

https://nexus.kxi-dev.kx.com/repository/{package.repo}/{package.name}/{package.version}/{package.name}-{package.version}.kxi"

Configuration of additional dependencies options can be done within the ~/.insights/pakx-config config which allows for modifications to the retrieval locations or added sources.

Local Dependencies

When adding local package dependencies:

path should be the absolute filepath
Specifying name and version and location: local is allowed
Searching path ordering is generally . followed by KXI_PACKAGE_PATH and finally KXI_ARTIFACT_PATH

The following is an example request which would find test-package.1.0.0.kxi based on a tagged release 1.0.0 stored at the location path/to/ on the local host.

N.B deps have location: local by default

PathName & Version

deps:
    - path: path/to/test-package-1.0.0.kxi

export KX_ARTIFACT_PATH=/path/to

deps:
    - name: test-package
      version: 0.0.1

Note

The version in the yaml will take precedence over the version in the filepath

Creating a package

Initializing a package

Package contents

Reserved Files And Directories

Converting Assemblies to Packages

Package Structure

Manifest File

Entrypoints

Package Dependencies

Installation

Dependency format

Local Dependencies

Next steps