kdb Insights Enterprise Packaging
Introduction
A kdb Insights Enterprise package is a group of components that can be managed as one entity via the kdb Insights Enterprise UI, or from outside kdb Insights Enterprise using the kxi package command in the kdb Insights CLI.
A package is made up of:
- A manifest file describing the content of a package and its dependencies.
- Deployment Components including details on how processes need to be created and orchestrated.
- Runtime Components defining the runtime behavior when you load a package from a q or python session.
- Static Context describing relationships with other packages.
- An initialization script defining the file/files to be loaded when a package is loaded.
A package is a directory containing files and sub-folders that describe these components.
Advantages of using packages over the deprecated assembly deployment mechanism.
The CLI and the kdb Insights Enterprise UI both interact with packages stored and deployed in kdb Insights Enterprise ensuring that the packages UI is kept in sync with the CLI and vice versa.
Package life cycle
The following list outlines the normal life cycle of a package, from creation to removal:
- Create a package locally.
- Add all the components you require to the local package.
- Pack the package locally into a
.kxi
package file, ready for upload to kdb Insights Enterprise. - Upload the
.kxi
package to kdb Insights Enterprise. - Deploy the package on kdb Insights Enterprise.
- Tear down the package on kdb Insights Enterprise.
- Remove a local package or a package on kdb Insights Enterprise.
Converting assemblies to packages
The CLI provides a conversion function to help you migrate from assemblies to packages. The remote-convert
function takes an assembly file and creates a package with the same name. The packages is stored in a directory next to the original assembly.
This new package can be managed via the CLI or the UI, once uploaded to a kdb Insights Enterprise deployment. When you deploy the package, it behaves in the same way as the assembly it was created from:
kxi package remote-convert --help
Usage: kxi package remote-convert [OPTIONS] ASSEMBLY_FILE
Convert V1/V2 assemblies into Packages with the server. Optionally keep the
package installed on the server.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --hostname,--url TEXT Insights URL [env var: INSIGHTS_URL] │
│ --realm TEXT Realm │
│ [env var: INSIGHTS_REALM] │
│ [default: insights] │
│ --client-id TEXT Client id │
│ [env var: INSIGHTS_CLIENT_ID] │
│ [default: test-publisher] │
│ --client-secret TEXT Client secret │
│ [env var: INSIGHTS_CLIENT_SECRET] │
│ --auth-enabled/--auth-disabled Will attempt to retrieve bearer │
│ token for request │
│ [env var: KXI_AUTH_ENABLED] │
│ [default: auth-enabled] │
│ --install Additionally installs the resulting │
│ package on the server. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
The example below runs the conversion of an existing assembly that includes a database, tables and a pipeline:
kxi package -q remote-convert asm.yaml && kxi package unpack asm && kxi package info asm
==PATH==
/Users/kx/packages/asm
==OBJECT==
Package
==Manifest==
name: asm
version: 0.0.1
databases:
asm-db:
dir: databases/asm-db
shards:
- asm-db-shard
tables:
- trade
- quote
- exchange
- instrument
pipelines:
sp-asm:
file: pipelines/sp-asm.yaml
router: router/router.yaml
deployment_config: deployment_config/deployment_config.yaml
If the conversion fails, kdb Insights Enterprise indicates which fields are problematic.
To ensure a successful conversion, modify the existing assembly by fixing the required fields. For example:
spec -> elements -> sm -> tiers -> 1 -> compression -> level field required (type=value_error.missing)
spec -> elements -> sm -> tiers -> 2 -> compression -> level field required (type=value_error.missing)
spec -> elements -> sm -> stream
extra fields not permitted (type=value_error.extra)
In the example above, the conversion requires the field compression to be added to the two tiers and the field "stream" to be removed from the Storage Manager.