Skip to content

Installing User Defined Analytics

User Defined Analytics (UDAs) allow you to define new APIs for databases to enable data querying and aggregation.

This section explains how to deploy a UDA in kdb Insights SDK that counts the number of records for specified columns and tables within a selected time range, as defined in the Creating a UDA and How to add a UDA to a package documentation.

Info

Users who need to call UDAs must have the insights.query.custom permission.

Prerequisites

Before deploying a package containing a UDA, ensure the following prerequisites are met:

Preparing the UDA

  1. Prepare the UDA package by following the UDA creation guide.

  2. Add the UDA to a package following the How to add a UDA to a package guide.

Note

If you are unfamiliar with how packaging works in kdb Insights Enterprise, review the Packaging Overview before continuing. This foundational knowledge is essential for understanding how UDAs are packaged and deployed.

Ensure the package is loaded

  1. Set the necessary environment variables for each component to locate and load the package:

Each component loading custom code from a package must have the KXI_PACKAGES and KX_PACKAGE_PATH environment variables set.

env:
  - name: KXI_PACKAGES
    value: "mypackage"
  - name: KX_PACKAGE_PATH
    value: "/opt/kx/packages"

Mount the package

  1. Mount the package as a volume to the folder specified in KX_PACKAGE_PATH.

Use -v to supply a volume:

docker run -e KX_PACKAGE_PATH=/opt/kx/packages\
  -e KXI_PACKAGES="mypackage"\
  -v /path/to/package:/opt/kx/packages

Set volumes and environment:

services:
  rdb:
    image: dap
    command: -p 5000
    environment:
      - KXI_PACKAGES=mypackage:1.0.0
      - KX_PACKAGE_PATH=/opt/kx/packages/
    volumes:
      - /path/to/package:/opt/kx/packages

Mount a volume under the container:

hostPath is used an example. This may be a persistent volume of any type.

spec:
  spec:
    containers:
      - name: dap
        image: dap
        env:
          - name: KXI_PACKAGES
            value: "mypackage:1.0.0"
          - name: KX_PACKAGE_PATH
            value: "/opt/kx/packages"
        volumeMounts:
        - mountPath: /opt/kx/packages
          name: mypackage-mount
    volumes:
      - name: mypackage-mount
        hostPath:
          path: /opt/kx/packages

Load the UDA

You can load a UDA without restarting core components like Data Access Processes (DAPs) and Aggregators (AGGs). This allows for faster development and testing.

To load a UDA, follow the instructions below:

  1. Expose the DAPs and AGG ports in compose.yaml. For example:

      kxi-da:
    ports:
      - 5081:5081
      - 5082:5082
      - 5083:5083
  kxi-agg:
    ports:
      - 5060:5060

    Ensure these ports are exposed to the client through port-forwarding before running these curl commands

  2. Define the curl endpoints. For example:

    export PKG=uda-pkg
export DAP=data-access
export AGG=aggregator
curl -X POST "http://localhost:5081/packages/post/load?package=$PKG&entry=$DAP"
curl -X POST "http://localhost:5082/packages/post/load?package=$PKG&entry=$DAP"
curl -X POST "http://localhost:5083/packages/post/load?package=$PKG&entry=$DAP"
curl -X POST "http://localhost:5060/packages/post/load?package=$PKG&entry=$AGG"

Example workflow

This example demonstrates how to load a UDA and execute it without redeploying the assembly, following the steps below:

  1. Start assembly running with no UDA
  2. Define UDA
  3. Load UDA into running DAPs and aggregator (no redeploy)
  4. Execute UDA

In this case, the UDA calculates the Open-High-Low-Close (OHLC) values from a trade table, which means that the function returns the first, maximum, minimum, and last prices.

Assembly running with no UDA

This example assumes you have an assembly running with no UDAs loaded.

  1. Start the assembly and insert some trade data:

    $ cd rt/
$ q startq.q
params:(`path`stream`publisher_id`cluster)!("/tmp/rt";"data";"pub1";enlist(":127.0.0.1:5002"))
p:.rt.pub params
n:20;
demodata:asc ([] time: .z.D+n?.z.T; id: n?`AAA`BBB; price:n?100.0; size: n?1000);
p(`.b; `trade; demodata)

  2. Verify the data with getData:

    h:hopen 5050
res:h(`.kxi.getData;enlist[`table]!enlist`trade;`;()!())
res[1]

    This returns an output similar to the below:

    time                          id  price    size
-----------------------------------------------
2025.08.27D01:09:50.253000000 AAA 63.46716 360
2025.08.27D01:23:46.281000000 AAA 94.41671 580
2025.08.27D02:13:45.275000000 AAA 23.92341 934
2025.08.27D02:23:53.831000000 AAA 23.06385 257
2025.08.27D02:31:11.038000000 AAA 93.67503 221
2025.08.27D02:57:02.433000000 AAA 57.59051 90
2025.08.27D04:04:03.440000000 BBB 38.9056  869
2025.08.27D05:50:50.463000000 BBB 27.82122 694
2025.08.27D06:16:08.787000000 BBB 96.72398 522
2025.08.27D06:28:56.306000000 AAA 43.9081  585
2025.08.27D07:14:17.624000000 AAA 84.81567 90
2025.08.27D07:46:26.915000000 AAA 47.07883 908
2025.08.27D08:04:43.016000000 AAA 94.9975  858
2025.08.27D08:23:27.554000000 AAA 59.19004 683
2025.08.27D08:48:23.285000000 BBB 8.123546 959
2025.08.27D08:50:31.645000000 AAA 97.85    997
2025.08.27D09:34:52.869000000 BBB 39.1543  468
2025.08.27D10:23:23.174000000 BBB 70.43314 314
2025.08.27D10:25:30.322000000 AAA 15.08133 865
2025.08.27D11:13:10.554000000 AAA 15.67317 344

  3. Check which APIs/UDAs are currently available:

    h:hopen 5050
getmeta:h(`.kxi.getMeta;()!();`;()!())
getmeta[1]`api

    Sample output:

    api            region aggFn           custom full metadata                                       ..
-------------------------------------------------------------------------------------------------..
.example.daAPI us     .example.aggAPI 1      1    `package`description`params`return`misc`aggRetu..
.kxi.getData   us     .sgagg.getData  0      1    `package`description`params`return`misc`aggRetu..
.kxi.ping      us                     0      1    `package`description`params`return`misc`aggRetu..
.kxi.preview   us     .sgagg.preview  0      1    `package`description`params`return`misc`aggRetu..
.kxi.qsql      us                     0      1    `package`description`params`return`misc`aggRetu..
.kxi.sql       us     .sgagg.sql      0      1    `package`description`params`return`misc`aggRetu..
.kxi.sql2      us     .sgagg.sql2     0      1    `package`description`params`return`misc`aggRetu..

Define UDA

Define the new UDA to calculate OHLC, .newuda.ohlc, of trade table:

manifest.yaml: 

cat ./config/packages/uda-pkg/0.0.1/manifest.yaml
name: uda-pkg
version: 0.0.1
entrypoints:
  data-access: src/uda.q
  aggregator: src/uda.q

src/uda.q: 

cat ./config/packages/uda-pkg/0.0.1/src/uda.q
.newuda.da:{[table;startTS;endTS]
    args:`table`startTS`endTS!(table;startTS;endTS);
    res:.kxi.selectTable args;
    .kxi.response.ok res
    };

.newuda.agg:{[tbls]
    res: select O:first price, H:max price,  L:min price,  C:last price by id from raze tbls;
    .kxi.response.ok res
    };

metadata:.kxi.metaDescription["OHLC UDA"],
    .kxi.metaParam[`name`type`isReq`description!(`table;-11h;1b;"Table to query")],
    .kxi.metaReturn[`type`description!(98h;"OHLC")],
    .kxi.metaMisc[enlist[`safe]!enlist 1b]

.kxi.registerUDA `name`query`aggregation`metadata!(`.newuda.ohlc;`.newuda.da;`.newuda.agg;metadata);

Load UDA

Load the UDA package into the running DAPs and aggregator, by defining the curl points, as follows:

export PKG=uda-pkg
export DAP=data-access
export AGG=aggregator

curl -X POST "http://localhost:5081/packages/post/load?package=$PKG&entry=$DAP"
curl -X POST "http://localhost:5082/packages/post/load?package=$PKG&entry=$DAP"
curl -X POST "http://localhost:5083/packages/post/load?package=$PKG&entry=$DAP"
curl -X POST "http://localhost:5060/packages/post/load?package=$PKG&entry=$AGG"

Execute UDA

Confirm and run the UDA, as follows:

  1. Check with getMeta:

    h:hopen 5050
getmeta:h(`.kxi.getMeta;()!();`;()!())
getmeta[1]`api

    Sample output showing the APIs/UDAs:

    api            region aggFn           custom full metadata                                       ..
-------------------------------------------------------------------------------------------------..
.example.daAPI us     .example.aggAPI 1      1    `package`description`params`return`misc`aggRetu..
.kxi.getData   us     .sgagg.getData  0      1    `package`description`params`return`misc`aggRetu..
.kxi.ping      us                     0      1    `package`description`params`return`misc`aggRetu..
.kxi.preview   us     .sgagg.preview  0      1    `package`description`params`return`misc`aggRetu..
.kxi.qsql      us                     0      1    `package`description`params`return`misc`aggRetu..
.kxi.sql       us     .sgagg.sql      0      1    `package`description`params`return`misc`aggRetu..
.kxi.sql2      us     .sgagg.sql2     0      1    `package`description`params`return`misc`aggRetu..
.newuda.ohlc   us     .newuda.agg     1      1    `package`description`params`return`misc`aggRetu..

  2. Execute the UDA:

    res:h(`.newuda.ohlc;enlist[`table]!enlist`trade;`;()!())
res[1]

    Sample output: 

    id | O        H        L        C
---| -----------------------------------
AAA| 63.46716 97.85    15.08133 15.67317
BBB| 38.9056  96.72398 8.123546 70.43314

Test the UDA

To test the UDA, refer to the example UDAs documentation for examples of queries.