Manage Runtime Components
This page describes how to manage the runtime components in kdb Insights Enterprise packages.
The runtime components within a package determine the behavior when you load a package from a q or Python session for tasks like testing the package in a scratchpad or testing user-defined functions.
Entrypoint components
Entrypoints define the q/Python files which are used as the initialization script for the components of your package that can load code. When loading a package using q or Python, entrypoints provide a method by which you can specify the sub-sections of your package code to be loaded into specific components.
The default entrypoint used when loading a package is default
and is defined as init.q
. This file is loaded by the components when no specific entrypoint defined.
You can update this entrypoint to be any file relative to the package root as shown below:
entrypoints:
default: src/init.q
You can also specify additional entrypoints for a package, allowing you to define specific code files to be loaded by different components of the deployment independently.
component | entrypoint name |
usage |
---|---|---|
Aggregator | aggregator |
Add an aggregation function that is part of a UDA |
Data Access Process | data-access |
Define a Query function that is part of a UDA |
Resource Coordinator | resource-coordinator |
Refer to the Resource Coordinator section for more information |
Storage Manager | storage-manager |
Refer to Storage configuration for more information |
The value of the entrypoint name
is important.
This name needs to be used as the value of --name
when calling the package add entrypoint
command
Though you can include arbitrarily named entrypoints (e.g. myentry
), only the above set of entrypoints are used by default by kdb Insights Enterprise.
Any component that does not have its own entrypoint in the manifest file loads the default
entrypoint.
The entrypoints named data-access
and aggregator
are used when defining User Defined Analytics (UDAs) in kdb Insights Enterprise. These entrypoints determine the code that is loaded by the Data Access and Aggregator processes respectively.
Add entrypoints
You can add entrypoints to a package using the kxi package add --to somepkg entrypoint --name myEntry
command. To add entrypoints for the Data Access and Aggregator processes you can run the following:
kxi package add --to mypkg entrypoint --name aggregator --path src/agg.q
kxi package add --to mypkg entrypoint --name data-access --path src/da.q
Running the kxi package add
commands above updates the entrypoints section of the manifest as follows:
entrypoints:
default: init.q
data-access: src/da.q
aggregator: src/agg.q
That can be loaded explicitly by name using the q or Python import library.
Within the Python and q package APIs it is possible to load these entrypoints separately.
-
Load the default entrypoint for version
1.0.0
of a package namedtest_pkg
:import kxi.packages kxi.packages.init() kxi.packages.load("test_pkg", "1.0.0")
-
Load the non-default entrypoint
myentry
for the same package version and name:import kxi.packages kxi.packages.init() kxi.packages.load("test_pkg", "1.0.0", "myentry")
-
Load the default entrypoint for version
1.0.0
of a package namedtest_pkg
:q).kxi.packages.load["test_pkg";"1.0.0"]
-
Load the non-default entrypoint
myentry
for the same package version and name:q).kxi.packages.load["test_pkg";"1.0.0";"myentry"]
Associating entrypoints with a database
To apply a UDA to a database you need to add environment variables flags to package. This ensures the entrypoints in the manifest file are loaded by the components. A simple way to do this is to add a deployment-config
which will set these env vars across all processes.
To apply a UDA when adding a database to a package, follow these steps:
-
Add a database to the package:
export PKG=mypackage kxi package init $PKG --force kxi package add --to $PKG database --name mydb kxi package info $PKG
==PATH== /builds/kxdev/documentation/insights/mypackage ==OBJECT== Package ==Manifest== uuid: a2238248-2cab-42b2-9761-88f2412db292 name: mypackage version: 0.0.1 metadata: description: '' authors: - {} entrypoints: default: init.q databases: mydb: dir: databases/mydb shards: - mydb-shard tables: []
-
Add an entrypoint for the
data-access
process:kxi package add --to $PKG entrypoint --name data-access --path src/da.q kxi package info $PKG
==PATH== /builds/kxdev/documentation/insights/mypackage ==OBJECT== Package ==Manifest== uuid: a2238248-2cab-42b2-9761-88f2412db292 name: mypackage version: 0.0.1 metadata: description: '' authors: - {} entrypoints: default: init.q data-access: src/da.q databases: mydb: dir: databases/mydb shards: - mydb-shard tables: []
-
Add a Deployment Configuration to a package:
kxi package add --to $PKG deployment-config kxi package info $PKG
==PATH== /builds/kxdev/documentation/insights/mypackage ==OBJECT== Package ==Manifest== uuid: a2238248-2cab-42b2-9761-88f2412db292 name: mypackage version: 0.0.1 metadata: description: '' authors: - {} entrypoints: default: init.q data-access: src/da.q databases: mydb: dir: databases/mydb shards: - mydb-shard tables: [] deployment_config: deployment_config/deployment_config.yaml
-
Update the
deployment_config.yaml
file to include the environment variable:echo "env: [{name: KXI_PACKAGES, value: ${PKG}:0.0.1}]" >> $PKG/deployment_config/deployment_config.yaml kxi package info $PKG
==PATH== /builds/kxdev/documentation/insights/mypackage ==OBJECT== Package ==Manifest== uuid: a2238248-2cab-42b2-9761-88f2412db292 name: mypackage version: 0.0.1 metadata: description: '' authors: - {} entrypoints: default: init.q data-access: src/da.q databases: mydb: dir: databases/mydb shards: - mydb-shard tables: [] deployment_config: deployment_config/deployment_config.yaml
Entrypoint fields
The fields for an entrypoint are described in the following table.
Field | Description |
---|---|
name | This field specifies the name of the entrypoint. It is optional and can be used to give a meaningful name to the entrypoint. |
filepath | This field specifies the path to the entrypoint file. |
The kxi package field info Entrypoint
command provides more details on the fields of an entrypoint, as shown below:
kxi package field info Entrypoint
Fields for Entrypoint
╭───────────────────────────────┬───────────────────────────────┬────────────────────────┬───────────────────────────────────────╮
│ field │ required │ type │ description │
├───────────────────────────────┼───────────────────────────────┼────────────────────────┼───────────────────────────────────────┤
│ name │ False │ string │ │
│ filepath │ False │ path │ │
╰───────────────────────────────┴───────────────────────────────┴────────────────────────┴───────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
Functions components
In kdb Insights Enterprise User Defined Functions (UDFs) allow you to deploy named functions written in q or Python within a package to a stream processor for use in a pipeline. For more information, refer to the documentation on how to manage user-defined functions (UDFs) within a package.
UDF fields
Use the kxi package field info UDF
command to see the fields for a UDF, as follows:
kxi package field info UDF
Fields for UDF
╭────────────────┬────────────┬───────────────────┬──────────────────────────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├────────────────┼────────────┼───────────────────┼──────────────────────────────────────────────────────────────────────────────┤
│ name │ True │ string │ Name of the UDF with no spaces │
│ function │ True │ string │ Native function name, inferred from code │
│ language │ True │ string │ Language, inferred from the file extension (py or q) │
│ file_path │ True │ string │ Location of the filepath within the package, inferred from context │
│ udf_sym │ True │ string │ The udf namespace this function is stored under with no spaces │
│ description │ False │ string │ A description of the udf │
│ category │ False │ List of string │ The categories of the udf, list or str │
│ tag │ False │ string │ Tag for the udf e.g. 1.0.0 │
╰────────────────┴────────────┴───────────────────┴──────────────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
Calling the package add
command removes an existing component if it shares the same type and name as the new component.
If you run the following command, the original copy of the mypkg
database is overwritten:
kxi package add database --to mypkg
# some editing of the files
kxi package add database --to mypkg
Remove components
Use the kxi package rm
command to remove components, as shown in the example below.
kxi package rm --help
Usage: kxi package rm [OPTIONS] COMMAND [ARGS]...
Remove an entity from the specified package.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --from -f <Package> Package from which the entity will be removed. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ For local development ──────────────────────────────────────────────────────╮
│ table Remove a table from the specified package │
│ database Remove a database from the specified package │
│ pipeline Remove a pipeline from the specified package. │
│ dep Remove a pipeline from the specified package. │
│ router Remove a router from the specified package. │
│ entrypoint Remove entrypoint from the specified package. │
│ deployment-config Remove deployment_config form the specified package │
│ patch Remove deployment_config form the specified package │
│ stream Remove stream from the specified package │
│ agg Remove aggregator from the specified package │
│ rc Remove resource_coordinator from the specified package │
│ shard Remove a shard from the specified package │
│ view Remove a View from the specified package. │
╰──────────────────────────────────────────────────────────────────────────────╯