How to Add a UDA to a Package

Introduction

Adding a User Defined Analytics (UDA) to a package involves organizing the necessary files, defining metadata, and configuring settings to ensure smooth deployment within kdb Insights.

Steps to add a UDA to a package

This page covers creating a package, adding a UDA to the package, and uploading the package to kdb Insights.

Prerequisites

Ensure that you have access to the kdb Insights environment.
Familiarize yourself with the basics of creating and managing packages in kdb Insights. For more information on packages, refer to the Package Lifecycle documentation.

Step 1 - Create a new package

Create a new package directory
Create a directory for your package. For example, name the directory myPackage.
```
mkdir myPackage
cd myPackage
```
Initialize the package
Use the kxi package init command to create a bare package structure.
```
kxi package init --artifact-store myPackage
```
Create the package structure
Within the myPackage directory, create a src folder if it does not already exist.
```
mkdir -p src
```

Step 2 - Create, define, and register the UDA

Create the custom UDA file
Inside the src directory, create a .q file for your UDA. For example, create myUDA.q.
```
touch src/myUDA.q
```
Define query and aggregation functions

Define your query and aggregation functions in myUDA.q. For more information on creating query and aggregation functions, refer to the UDA creating documentation. Here are examples of each:

Example - Aggregation Function

q .example.aggFn:{[res] .kxi.response.ok 100?raze res };

Example - Query Functions

Arguments as Values

.example.queryFn:{[table;startTS;endTS;columns]
   columns:$[-11h = type columns;enlist columns;columns];
   data:.kxi.selectTable`table`startTS`endTS`agg!(table;startTS;endTS;columns!columns); // Retrieve data within specified time range
 .kxi.response.ok data
 };

Arguments as a Dictionary

.example.queryFn:{[args]
    columns:$[-11h = type columns:args`columns;enlist columns;columns]
    filter:enlist (<;`i;100) // Note: For partitioned tables, returns the first 100 per date
    .kxi.response.ok ?[args`table;filter;0b;columns!columns]
 };

Register the UDA
Register the UDA from the Service Gateway using the registerUDA function. For example:
```
registerUDA[`myUDA; `queryFn; `aggFn]
```
The registerUDA function associates the query function queryFn and the aggregation function aggFn with the UDA myUDA.
Update the package metadata

Make sure your package metadata file includes a reference to the UDA file. For example, add the following entry:

{
    "uuid": "8ed7b25f-9e5c-4c03-a1bd-480047995fcd",
    "name": "da-ext",
    "version": "0.0.1",
    "dependencies": [],
    "metadata": {
        "authors": [
            {
                "name": "Your Name"
            }
        ]
    },
    "entrypoints": {
        "aggregator": "agg.q",
        "resource-coordinator": "rc.q",
        "data-access": "da.q",
        "storage-manager": "sm.q",
        "profiler": "profiler.q"
    },
    "license": ""
}

Step 3 - Build the package

Build the package
Use the kxi package packit command to create a package artifact from your source code directory.
```
kxi package packit --artifact-store myPackage
```

Step 4 - Upload and test the package

Authenticate with kdb Insights:
Log in to your kdb Insights environment. For more information on CLI authentication, refer to the Authentication documentation. Run the following command:
```
kxi auth login
```
Upload the package:
Use the kxi package push command to upload your package. Replace myPackage.q with the path to your package file.
```
kxi package push myPackage.q
```
Info

The query function is included in the DAPs within the package, but it is not available in the global Agg. To allow the global Agg to access the query function, deploy an Agg with your package or update the package on the global Agg and restart it after deployment. Restarting the global Agg requires cluster access.
Test the UDA
Load the package in your kdb Insights environment and test the UDA. Connect to your kdb Insights environment and run:

qREST

Gateway URL

The GATEWAY variable below is defined as an IPC connection to the Service Gateway. For example, :insights-qe-gateway:5050 connects to the query environment gateway within an insights namespace.

args:`table`startTS`endTS`byCols!(`trade;"p"$.z.D-3;"p"$.z.D-2;`date`sym)

GATEWAY (`.custom.countBy; args; `; ()!())

Verify the UDA produces the expected results. Refer to the following example with a successful header and payload response:

show each GATEWAY(`.custom.countBy;args;`;()!())
rcvTS     | 2024.08.26D18:30:58.928000000
corr      | 30d4b5e5-47dd-4bd0-bd37-20ffb498486f
logCorr   | "30d4b5e5-47dd-4bd0-bd37-20ffb498486f"
api       | `.custom.countBy
agg       | `:10.244.0.128:5070
refVintage| 2024082600000000054
rc        | 0h
ac        | 0h
ai        | ""

sym       date      | cnt
--------------------| ----
DNDF.CAN  2024.08.23| 3489
FNLH.CAN  2024.08.23| 3470
JBMH.CAN  2024.08.23| 3502
JFMB.CAN  2024.08.23| 3408
MGPJ.CAN  2024.08.23| 3411
....

Gateway URL

The $GATEWAY variable should point at your kdb Insights install. For a microservice install, this will be the hostname of the install using port 8080. For an enterprise install, this is your $INSIGHTS_HOSTNAME with /servicegateway as the URL prefix.

  curl -X POST "$GATEWAY/your/uda" \
      -H "Content-Type: application/json" \
      -H "Accept: application/json" \
      -H "Authorization: Bearer $INSIGHTS_TOKEN" \
      -d "$(jq -n \
          '{
              ... # Your parameters here
          }' | jq -cr .)"

Verify that the UDA produces the expected results. Refer to the following example with a successful header and payload response:

    {
  "header": {
    "rcvTS": "2024-08-26T19:13:35.760000000",
    "corr": "f6a40872-0dcf-4b0b-a881-f3b12868d297",
    "logCorr": "f6a40872-0dcf-4b0b-a881-f3b12868d297",
    "http": "json",
    "api": ".custom.countBy",
    "agg": ":10.244.0.128:5070",
    "refVintage": 2024082600000000054,
    "rc": 0,
    "ac": 0,
    "ai": ""
  },
  "payload": [
    {
      "sym": "CLFC.USAN",
      "date": "2024-08-23",
      "cnt": 3551
    },
    {
      "sym": "CNLC.USAN",
      "date": "2024-08-23",
      "cnt": 3464
    },
    {
      "sym": "CPEL.USAN",
      "date": "2024-08-23",
      "cnt": 3380
    },
   ...

Handling Unsuccessful Deployments

Identify Errors

If the deployment fails, review error messages and logs to identify the cause. Common issues include missing dependencies, incorrect configurations, or permission errors.

Roll Back Changes

If necessary, roll back the deployment to the last known good state to avoid disruptions in production.

Troubleshoot and Redeploy

Troubleshoot the identified issues, make the necessary corrections, and attempt the deployment again.