Skip to content

Manage Runtime Components

This page describes how to manage the runtime components in kdb Insights Enterprise packages.

The runtime components within a package determine the behavior when you load a package from a q or Python session for tasks like testing the package in a scratchpad or testing user-defined functions.

Entrypoint components

Entrypoints define the q/Python files which are used as the initialization script for the components of your package that can load code. When loading a package using q or Python, entrypoints provide a method by which you can specify the sub-sections of your package code to be loaded into specific components.

The default entrypoint used when loading a package is default and is defined as init.q. This file is loaded by the components when no specific entrypoint defined.

You can update this entrypoint to be any file relative to the package root as shown below:

entrypoints:
    default: src/init.q

You can also specify additional entrypoints for a package, allowing you to define specific code files to be loaded by different components of the deployment independently.

component entrypoint name usage
Aggregator aggregator Add an aggregation function that is part of a UDA
Data Access Process data-access Define a Query function that is part of a UDA
Resource Coordinator resource-coordinator Refer to the Resource Coordinator section for more information
Storage Manager storage-manager Refer to Storage configuration for more information

The value of the entrypoint name is important.

This name needs to be used as the value of --name when calling the package add entrypoint command

Though you can include arbitrarily named entrypoints (e.g. myentry), only the above set of entrypoints are used by default by kdb Insights Enterprise. Any component that does not have its own entrypoint in the manifest file loads the default entrypoint.

The entrypoints named data-access and aggregator are used when defining User Defined Analytics (UDAs) in kdb Insights Enterprise. These entrypoints determine the code that is loaded by the Data Access and Aggregator processes respectively.

Add entrypoints

You can add entrypoints to a package using the kxi package add --to somepkg entrypoint --name myEntry command. To add entrypoints for the Data Access and Aggregator processes you can run the following:

kxi package add --to mypkg entrypoint --name aggregator --path src/agg.q
kxi package add --to mypkg entrypoint --name data-access --path src/da.q

Running the kxi package add commands above updates the entrypoints section of the manifest as follows:

entrypoints:
    default: init.q
    data-access: src/da.q
    aggregator: src/agg.q

That can be loaded explicitly by name using the q or Python import library.

Within the Python and q package APIs it is possible to load these entrypoints separately.

  • Load the default entrypoint for version 1.0.0 of a package named test_pkg:

    import kxi.packages 
    kxi.packages.init()
    kxi.packages.load("test_pkg", "1.0.0")
    
  • Load the non-default entrypoint myentry for the same package version and name:

    import kxi.packages
    kxi.packages.init()
    kxi.packages.load("test_pkg", "1.0.0", "myentry")
    
  • Load the default entrypoint for version 1.0.0 of a package named test_pkg:

    q).kxi.packages.load["test_pkg";"1.0.0"]
    
  • Load the non-default entrypoint myentry for the same package version and name:

    q).kxi.packages.load["test_pkg";"1.0.0";"myentry"]
    

Associating entrypoints with a database

To apply a UDA to a database you need to add environment variables flags to package. This ensures the entrypoints in the manifest file are loaded by the components. A simple way to do this is to add a deployment-config which will set these env vars across all processes.

To apply a UDA when adding a database to a package, follow these steps:

  1. Add a database to the package:

    export PKG=mypackage
    kxi package init $PKG --force
    kxi package add --to $PKG database --name mydb 
    kxi package info $PKG
    
    ==PATH==
    /builds/kxdev/documentation/insights/mypackage
    
    ==OBJECT==
    Package
    
    ==Manifest==
    uuid: a2238248-2cab-42b2-9761-88f2412db292
    name: mypackage
    version: 0.0.1
    metadata:
      description: ''
      authors:
      - {}
    entrypoints:
      default: init.q
    databases:
      mydb:
        dir: databases/mydb
        shards:
        - mydb-shard
        tables: []
    

  2. Add an entrypoint for the data-access process:

    kxi package add --to $PKG entrypoint --name data-access --path src/da.q
    kxi package info $PKG
    
    ==PATH==
    /builds/kxdev/documentation/insights/mypackage
    
    ==OBJECT==
    Package
    
    ==Manifest==
    uuid: a2238248-2cab-42b2-9761-88f2412db292
    name: mypackage
    version: 0.0.1
    metadata:
      description: ''
      authors:
      - {}
    entrypoints:
      default: init.q
      data-access: src/da.q
    databases:
      mydb:
        dir: databases/mydb
        shards:
        - mydb-shard
        tables: []
    

  3. Add a Deployment Configuration to a package:

    kxi package add --to $PKG deployment-config
    kxi package info $PKG
    
    ==PATH==
    /builds/kxdev/documentation/insights/mypackage
    
    ==OBJECT==
    Package
    
    ==Manifest==
    uuid: a2238248-2cab-42b2-9761-88f2412db292
    name: mypackage
    version: 0.0.1
    metadata:
      description: ''
      authors:
      - {}
    entrypoints:
      default: init.q
      data-access: src/da.q
    databases:
      mydb:
        dir: databases/mydb
        shards:
        - mydb-shard
        tables: []
    deployment_config: deployment_config/deployment_config.yaml
    

  4. Update the deployment_config.yaml file to include the environment variable:

    echo "env: [{name: KXI_PACKAGES, value: ${PKG}:0.0.1}]" >> $PKG/deployment_config/deployment_config.yaml
    kxi package info $PKG
    
    ==PATH==
    /builds/kxdev/documentation/insights/mypackage
    
    ==OBJECT==
    Package
    
    ==Manifest==
    uuid: a2238248-2cab-42b2-9761-88f2412db292
    name: mypackage
    version: 0.0.1
    metadata:
      description: ''
      authors:
      - {}
    entrypoints:
      default: init.q
      data-access: src/da.q
    databases:
      mydb:
        dir: databases/mydb
        shards:
        - mydb-shard
        tables: []
    deployment_config: deployment_config/deployment_config.yaml
    

Entrypoint fields

The fields for an entrypoint are described in the following table.

Field Description
name This field specifies the name of the entrypoint. It is optional and can be used to give a meaningful name to the entrypoint.
filepath This field specifies the path to the entrypoint file.

The kxi package field info Entrypoint command provides more details on the fields of an entrypoint, as shown below:

kxi package field info Entrypoint
 Fields for Entrypoint                                                                                                            
╭───────────────────────────────┬───────────────────────────────┬────────────────────────┬───────────────────────────────────────╮
│ field                         │ required                      │ type                   │ description                           │
├───────────────────────────────┼───────────────────────────────┼────────────────────────┼───────────────────────────────────────┤
│ name                          │ False                         │ string                 │                                       │
│ filepath                      │ False                         │ path                   │                                       │
╰───────────────────────────────┴───────────────────────────────┴────────────────────────┴───────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)

Functions components

In kdb Insights Enterprise User Defined Functions (UDFs) allow you to deploy named functions written in q or Python within a package to a stream processor for use in a pipeline. For more information, refer to the documentation on how to manage user-defined functions (UDFs) within a package.

UDF fields

Use the kxi package field info UDF command to see the fields for a UDF, as follows:

kxi package field info UDF
 Fields for UDF                                                                                                                   
╭────────────────┬────────────┬───────────────────┬──────────────────────────────────────────────────────────────────────────────╮
│ field          │ required   │ type              │ description                                                                  │
├────────────────┼────────────┼───────────────────┼──────────────────────────────────────────────────────────────────────────────┤
│ name           │ True       │ string            │ Name of the UDF with no spaces                                               │
│ function       │ True       │ string            │ Native function name, inferred from code                                     │
│ language       │ True       │ string            │ Language, inferred from the file extension (py or q)                         │
│ file_path      │ True       │ string            │ Location of the filepath within the package, inferred from context           │
│ udf_sym        │ True       │ string            │ The udf namespace this function is stored under with no spaces               │
│ description    │ False      │ string            │ A description of the udf                                                     │
│ category       │ False      │ List of string    │ The categories of the udf, list or str                                       │
│ tag            │ False      │ string            │ Tag for the udf e.g. 1.0.0                                                   │
╰────────────────┴────────────┴───────────────────┴──────────────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)

Calling the package add command removes an existing component if it shares the same type and name as the new component.

If you run the following command, the original copy of the mypkg database is overwritten:

kxi package add database --to mypkg
# some editing of the files
kxi package add database --to mypkg

Remove components

Use the kxi package rm command to remove components, as shown in the example below.

kxi package rm --help
 Usage: kxi package rm [OPTIONS] COMMAND [ARGS]...                              

 Remove an entity from the specified package.                                   

╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --from  -f  <Package>  Package from which the entity will be removed.        │
│ --help                 Show this message and exit.                           │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ For local development ──────────────────────────────────────────────────────╮
│ table               Remove a table from the specified package                │
│ database            Remove a database from the specified package             │
│ pipeline            Remove a pipeline from the specified package.            │
│ dep                 Remove a pipeline from the specified package.            │
│ router              Remove a router from the specified package.              │
│ entrypoint          Remove entrypoint from the specified package.            │
│ deployment-config   Remove deployment_config form the specified package      │
│ patch               Remove deployment_config form the specified package      │
│ stream              Remove stream from the specified package                 │
│ agg                 Remove aggregator from the specified package             │
│ rc                  Remove resource_coordinator from the specified package   │
│ shard               Remove a shard from the specified package                │
│ view                Remove a View from the specified package.                │
╰──────────────────────────────────────────────────────────────────────────────╯

Next steps