How to Add a UDA to a Package
Introduction
Adding a User Defined Analytics (UDA) to a package involves organizing the necessary files, defining metadata, and configuring settings to ensure smooth deployment within kdb Insights.
Steps to add a UDA to a package
This page covers creating a package, adding a UDA to the package, and uploading the package to kdb Insights.
Prerequisites
- Ensure that you have access to the kdb Insights environment.
- Familiarize yourself with the basics of creating and managing packages in kdb Insights. For more information on packages, refer to the Package Lifecycle documentation.
Step 1 - Create a new package
-
Create a new package directory
-
Create a directory for your package. For example, name the directory
myPackage
.mkdir myPackage cd myPackage
-
Initialize the package
-
Use the
kxi package init
command to create a bare package structure.kxi package init --artifact-store myPackage
-
Create the package structure
-
Within the
myPackage
directory, create asrc
folder if it does not already exist.mkdir -p src
Step 2 - Create, define, and register the UDA
-
Create the custom UDA file
-
Inside the
src
directory, create a.q
file for your UDA. For example, createmyUDA.q
.touch src/myUDA.q
-
Define query and aggregation functions
-
Define your query and aggregation functions in
myUDA.q
. For more information on creating query and aggregation functions, refer to the UDA creating documentation. Here are examples of each:Example - Aggregation Function
q .example.aggFn:{[res] .kxi.response.ok 100?raze res };
Example - Query Functions
Arguments as Values
.example.queryFn:{[table;startTS;endTS;columns] columns:$[-11h = type columns;enlist columns;columns]; data:.kxi.selectTable`table`startTS`endTS`agg!(table;startTS;endTS;columns!columns); // Retrieve data within specified time range .kxi.response.ok data };
Arguments as a Dictionary
.example.queryFn:{[args] columns:$[-11h = type columns:args`columns;enlist columns;columns] filter:enlist (<;`i;100) // Note: For partitioned tables, returns the first 100 per date .kxi.response.ok ?[args`table;filter;0b;columns!columns] };
-
Register the UDA
-
Register the UDA from the Service Gateway using the
registerUDA
function. For example:registerUDA[`myUDA; `queryFn; `aggFn]
-
The
registerUDA
function associates the query functionqueryFn
and the aggregation functionaggFn
with the UDAmyUDA
. -
Update the package metadata
-
Make sure your package metadata file includes a reference to the UDA file. For example, add the following entry:
{ "uuid": "8ed7b25f-9e5c-4c03-a1bd-480047995fcd", "name": "da-ext", "version": "0.0.1", "dependencies": [], "metadata": { "authors": [ { "name": "Your Name" } ] }, "entrypoints": { "aggregator": "agg.q", "resource-coordinator": "rc.q", "data-access": "da.q", "storage-manager": "sm.q", "profiler": "profiler.q" }, "license": "" }
Step 3 - Build the package
-
Build the package
-
Use the
kxi package packit
command to create a package artifact from your source code directory.kxi package packit --artifact-store myPackage
Step 4 - Upload and test the package
- Authenticate with kdb Insights:
-
Log in to your kdb Insights environment. For more information on CLI authentication, refer to the Authentication documentation. Run the following command:
kxi auth login
-
Upload the package:
-
Use the
kxi package push
command to upload your package. ReplacemyPackage.q
with the path to your package file.kxi package push myPackage.q
Info
The query function is included in the DAPs within the package, but it is not available in the global Agg. To allow the global Agg to access the query function, deploy an Agg with your package or update the package on the global Agg and restart it after deployment. Restarting the global Agg requires cluster access.
-
Test the UDA
-
Load the package in your kdb Insights environment and test the UDA. Connect to your kdb Insights environment and run:
Gateway URL
The GATEWAY
variable below is defined as an IPC connection to the Service Gateway. For example, :insights-qe-gateway:5050
connects to the query environment gateway within an insights
namespace.
args:`table`startTS`endTS`byCols!(`trade;"p"$.z.D-3;"p"$.z.D-2;`date`sym)
GATEWAY (`.custom.countBy; args; `; ()!())
- Verify the UDA produces the expected results. Refer to the following example with a successful header and payload response:
show each GATEWAY(`.custom.countBy;args;`;()!())
rcvTS | 2024.08.26D18:30:58.928000000
corr | 30d4b5e5-47dd-4bd0-bd37-20ffb498486f
logCorr | "30d4b5e5-47dd-4bd0-bd37-20ffb498486f"
api | `.custom.countBy
agg | `:10.244.0.128:5070
refVintage| 2024082600000000054
rc | 0h
ac | 0h
ai | ""
sym date | cnt
--------------------| ----
DNDF.CAN 2024.08.23| 3489
FNLH.CAN 2024.08.23| 3470
JBMH.CAN 2024.08.23| 3502
JFMB.CAN 2024.08.23| 3408
MGPJ.CAN 2024.08.23| 3411
....
Gateway URL
The $GATEWAY
variable should point at your kdb Insights install. For a microservice install, this will be the hostname of the install using port 8080. For an enterprise install, this is your $INSIGHTS_HOSTNAME
with /servicegateway
as the URL prefix.
curl -X POST "$GATEWAY/your/uda" \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer $INSIGHTS_TOKEN" \
-d "$(jq -n \
'{
... # Your parameters here
}' | jq -cr .)"
- Verify that the UDA produces the expected results. Refer to the following example with a successful header and payload response:
{
"header": {
"rcvTS": "2024-08-26T19:13:35.760000000",
"corr": "f6a40872-0dcf-4b0b-a881-f3b12868d297",
"logCorr": "f6a40872-0dcf-4b0b-a881-f3b12868d297",
"http": "json",
"api": ".custom.countBy",
"agg": ":10.244.0.128:5070",
"refVintage": 2024082600000000054,
"rc": 0,
"ac": 0,
"ai": ""
},
"payload": [
{
"sym": "CLFC.USAN",
"date": "2024-08-23",
"cnt": 3551
},
{
"sym": "CNLC.USAN",
"date": "2024-08-23",
"cnt": 3464
},
{
"sym": "CPEL.USAN",
"date": "2024-08-23",
"cnt": 3380
},
...
Handling Unsuccessful Deployments
Identify Errors
- If the deployment fails, review error messages and logs to identify the cause. Common issues include missing dependencies, incorrect configurations, or permission errors.
Roll Back Changes
- If necessary, roll back the deployment to the last known good state to avoid disruptions in production.
Troubleshoot and Redeploy
- Troubleshoot the identified issues, make the necessary corrections, and attempt the deployment again.