Code Examples and Templates

This page provides code examples and templates to accelerate your development process and ensure that your UDAs adhere to best practices.

Code Examples

1. Loading other file dependencies

As mentioned on the best practices page, you should break down complex analytics into smaller, reusable components. This makes your UDAs easier to manage and test.

If your smaller functions are located in separate files, you can load those files as follows:

// Custom UDA code.

.kxi.packages.file.load"common.q" / Example loading other file dependencies

2. Provide the first 100 records from each DAP

Define the Query function:
```
// DA query function
.example.queryFn:{[table;columns]
    columns:$[-11h = type columns;enlist columns;columns];
    filter:enlist (<;`i;100); // Note for partitioned tables, will return first 100 per date
    .kxi.response.ok ?[table;filter;0b;columns!columns]
    };
```
Points to consider
- The parameters are the the table and column names you wish to return.
- The filter is hard-coded to the first 100 rows and the columns field is converted into a list of strings.
- ? is used here instead of selectTable. This bypasses row-level entitlements, if they are enabled.

Add metadata:

// Metadata for registering
metadata:.kxi.metaDescription["Returns first 100 records of each DAP"], / Description of the UDA
.kxi.metaParam[`name`type`isReq`description!(`table;-11h;1b;"Name of the table to query.")], / Describe `table` parameter
.kxi.metaParam[`name`type`isReq`description!(`columns;11 -11h;1b;"Columns in table to select")], / Describe `columns` parameter
.kxi.metaReturn`type`description!(98h;"Return first 100 rows."), / Describe the return of the UDA
.kxi.metaMisc[enlist[`safe]!enlist 1b]; / States that this UDA is safe to be retried in the event of an error

Register the UDA:

// Registration
.kxi.registerUDA `name`query`metadata!(`.example.testUDA`.example.queryFn;metadata)

Note

No aggregation function is required as the raze function can be used.

Once the UDA is added to a package and deployed, the following requests can be made:

qREST

Gateway URL

The GATEWAY variable below is defined as an IPC connection to the Service Gateway. For example, `:insights-qe-gateway:5050 would connect to the query environment gateway within an insights namespace.

// args:`table`columns!("tablename;"column1,column2")
// show each GATEWAY(`.example.testUDA;args;`;()!())

Gateway URL

The $GATEWAY variable should point at the hostname of the install using port 8080.

TABLE="tablename"
COLUMNS="column1,column2"
curl -X POST --header "Content-Type: application/json"\
    --header "Accepted: application/json"\
    --data "{\"startTS\": \"$startTS\", \"endTS\": \"$endTS\"}"\
    "https://${INSIGHTS_HOSTNAME}/servicegateway/example/testUDA"

3. Provide summary values

This UDA provides a summary of total and average usage by the first 3 digits of postal codes in a table.

Example schema:

The data table readings contains information about sensor usage and the Postal Code of the sensor.

q)readings
sensorID readTS                        val postal
-------------------------------------------------
0        2024.08.12D17:32:37.645609220 100 P0P1K0
1        2024.08.12D17:32:37.645609220 200 K7K2C7
q)meta readings
c       | t f a
--------| -----
sensorID| j
readTS  | p
val     | f
postal  | s

Define the Query Function:

The usageQuery function computes the count and sum of the val column, grouped by the first three digits of the postal code:

// @desc Computes the count of records as well as the sum of `val` column by first 3 digits of postal code.
//
// @param startTS   {timestamp}     Starting timestamp.
// @param endTS     {timestamp}     Ending timestamp.
// @param filter    {list}          Filter to apply to data.
//
// @return         {table}         Average count by postal code.
//
.example.usageQuery:{[startTS;endTS;filter]
    args:`table`startTS`endTS`columns`filter`groupBy!(`readings;startTS;endTS;`sumVal`cntVal!((sum;`val);(count;`val));filter;enlist[`postal]!enlist`postal); / Build arguments
    res:.kxi.selectTable args; / Make selectTable call
    res:0!select sumVal:sum sumVal,cntVal:sum cntVal by postal:`$3#/:string postal from res; / Anonymize data, and perform further processing
    .kxi.response.ok res / Return with OK
    };

Points to consider

The parameters are the time range and the filter to be applied to the raw data.
selectTable returns the sum and count of the val column per postal code.
The records returned from selectTable are then summed and counted by the first 3 digits of postal code.

Define the Aggregation Function:

The usageAgg function consolidates partial results from DAPs and calculates the total usage statistics, including count and average per postal code:

// @desc Aggregates partial results from DAPs to produce total usage statistics, count of records, and average per postal code.
//
// @param tbls  {table[]}   List of tables with postal, sumVal, and cntVal columns.
//
// @return      {table}     Table with sum, count, and average by first 3 digits of postal code.
//
.example.usageAgg:{[tbls]
    .kxi.response.ok 0!update avgVal:sumVal%cntVal from select sumVal:sum sumVal, cntVal:sum cntVal by postal from raze tbls
    };

Points to note

There is no filter parameter as it is only needed by the query function.
The partials are merged using the raze method.
Once the partials are merged the update statement aggregates the sum and count values across the tiers per 3 digit postal code returned from the query function.

Add metadata:

Define metadata for the UDA to describe its purpose, parameters, and return values:

// Define metadata. 
metadata:.kxi.metaDescription["Returns the average by postal code across the first 100 records of each DAP"], / Description of the UDA
    .kxi.metaParam[`name`type`isReq`description!(`startTS;-12h;1b;"Start timestamp of query")], / Describe `startTS` parameter
    .kxi.metaParam[`name`type`isReq`description!(`endTS;-12h;1b;"End timestamp of query")], / Describe `endTS` parameter
    .kxi.metaParam[`name`type`isReq`default`description!(`filter;0h;0b;();"Optional filter to apply to readings table")], / Describe `filter` parameter
    .kxi.metaReturn`type`description!(98h;"The summary of total and average usage by first 3 digits of postal code"), / Describe the return of the UDA

Points to consider

The filter parameter is optional, all data in the table is used if it is not set.

Register the UDA:

Register the UDA with the appropriate names for query and aggregation functions, and provide the metadata:

.kxi.registerUDA `name`query`aggregation`metadata!(`.example.usageByPostalCode;`.example.usageQuery;`.example.usageAgg;metadata) / Register UDA

Once the UDA is deployed as part of a package the following requests can be made:

qREST

Gateway URL

The GATEWAY variable below is defined as an IPC connection to the Service Gateway. For example, `:insights-qe-gateway:5050 would connect to the query environment gateway within an insights namespace.

// args:`startTS`endTS!("p"$.z.D-3;"p".z.D-2)
// show each GATEWAY(`.example.usageByPostalCode;args;`;()!())

Gateway URL

The $GATEWAY variable should point at the hostname of the install using port 8080.

startTS=$(date -u '+%Y.%m.%dD%H:00:00')
endTS=$(date -u '+%Y.%m.%dD%H:%M%:%S')
curl -X POST --header "Content-Type: application/json"\
    --header "Accepted: application/json"\
    --data "{\"startTS\": \"$startTS\", \"endTS\": \"$endTS\"}"\
    "https://${$GATEWAY}/example/usageByPostalCode"

4. Join Trade and Quote tables

UDA definition:

\d .customUDA

// Join Trade and Quote tables

// Define query function.
//
// @desc Define a new API. Get trade data for specified date range.
// @param startTS   {timestamp}         Start time (inclusive).
// @param endTS     {timestamp}         End time (exclusive).
// @param s         {symbol|symbol[]}   Market data symbols to query for.
// @return          {table}             Count by specified columns.
queryMktData:{[startTS;endTS;syms]
    if[`~syms;
        :.kxi.response.error[.kxi.response.ac.INPUT;"Must enter a non null symbol(s) to query";()]];
    symFilt:(in;`sym;enlist syms);
    filter:((within;`realTime;(startTS;endTS));symFilt);
    t:.kxi.selectTable`table`startTS`endTS`filter!(`trade;startTS;endTS;filter);
    .kxi.response.ok t
    }

// Define aggregation function.
//
// @desc Get symbol and timerange for quote data, then defer the final result until we retrieve the quote data.
// @param trade {table}     Trade data from the query result.
aggMktData:{[trade]
    trade:raze trade; / Raze as data coming from multiple DAPs
    .kxi.context.set[`trade;trade]; / Save trade data in context
    symFilt:exec distinct sym from trade;
    args:`table`startTS`endTS`agg`sortCols!
        (`quote;"p"$"d"$min trade`realTime;1+max trade`realTime;
        `sym`realTime`ask`askSize`bid`bidSize;`sym`realTime); / Args for quote getData
    .kxi.response.callAPI[`.kxi.getData;args;`.customUDA.aggMktDataResume;()!()]
    }

// Aggregation defer resume.
// @desc    Resume deferred request and join together trade and quote data.
// @param   quote   {table}     Quote data.
// @return  Joined trade and quote data for specified syms with numeric values rounded to 2 decimal places.
aggMktDataResume:{[quote]
    trade:.kxi.context.get`trade; / Recover trade data from context
    res:aj[`sym`realTime;trade;quote]; / Join
    round:{("j"$100*x)%100}; / Round to two decimals
    .kxi.response.ok update round price,round bid,round ask from res
    }

\d .

// Define metadata.
metadata:.kxi.metaDescription["Custom UDA - join trade and quote data."],
    .kxi.metaMisc[enlist[`safe]!enlist 1b],
    .kxi.metaParam[`name`type`isReq`description!(`startTS;-12h;1b;"Start time (inclusive).")],
    .kxi.metaParam[`name`type`isReq`description!(`endTS;-12h;1b;"End time (exclusive).")],
    .kxi.metaParam[`name`type`isReq`description!(`syms;11 -11h;1b;"Sym values to query on. Must be a non-null value.")],
    .kxi.metaReturn`type`description!(98h;"Joined trade and quote data.");

// Registration.
.kxi.registerUDA `name`query`aggregation`metadata!(`.customUDA.getMktData;`.customUDA.queryMktData;`.customUDA.aggMktData;metadata);

Once the UDA is deployed as part of a package the following requests can be made:

qREST

Gateway URL

The GATEWAY variable below is defined as an IPC connection to the Service Gateway. For example, `:insights-qe-gateway:5050 would connect to the query environment gateway within an insights namespace.

// Example qIPC requests.
// GATEWAY variable below is defined as an IPC connection to the Service Gateway.
// args:`startTS`endTS`syms!("p"$.z.D-3;"p".z.D-2;`DNDF.CAN)
// show each GATEWAY(`.customUDA.getMktData;args;`;()!())

Gateway URL

The $GATEWAY variable should point at the hostname of the install using port 8080.

TABLE="tablename"
COLUMNS="column1,column2"
curl -X POST --header "Content-Type: application/json"\
    --header "Accepted: application/json"\
    --data "{\"table\": \"$TABLE\", \"columns\": \"$COLUMNS\"}"\
    "https://${$GATEWAY}/example/testUDA"

5. Provide Open, High, Low, and Close values

Define the new UDA to calculate OHLC of a trade table:

UDA definition:

//TO DO Mike to give me an we do the agg in the DAPs as well to reduce the data transfer?

.newuda.da:{[table;startTS;endTS]
    args:`table`startTS`endTS!(table;startTS;endTS);
    res:.kxi.selectTable args;
    .kxi.response.ok res
    };

.newuda.agg:{[tbls]
    res: select O:first price, H:max price,  L:min price,  C:last price by id from raze tbls;
    .kxi.response.ok res
    };

metadata:.kxi.metaDescription["OHLC UDA"],
    .kxi.metaParam[`name`type`isReq`description!(`table;-11h;1b;"Table to query")],
    .kxi.metaReturn[`type`description!(98h;"OHLC")],
    .kxi.metaMisc[enlist[`safe]!enlist 1b]

.kxi.registerUDA `name`query`aggregation`metadata!(`.newuda.ohlc;`.newuda.da;`.newuda.agg;metadata);

6. Include logging

This example adds logging to the First 100 records from each DAP example.

\d .example

.log.initns[];

// DA query function
.example.queryFn:{[table;columns]
    .uda.log.debug enlist[`table`columns!
        ("Starting .example.queryFn"; table; columns)];

    columns:$[-11h = type columns;enlist columns;columns];
    filter:enlist (<;`i;100); // Note for partitioned tables, will return first 100 per date
    .kxi.response.ok ?[table;filter;0b;columns!columns]

    / Log that the function is complete
    .uda.log.info ".example.queryFn: complete";

    };

metadata:.kxi.metaDescription["Returns first 100 records of each DAP"], / Description of the UDA
    .kxi.metaParam[`name`type`isReq`description!(`table;-11h;1b;"Name of the table to query.")], / Describe `table` parameter
    .kxi.metaParam[`name`type`isReq`description!(`columns;11 -11h;1b;"Columns in table to select")], / Describe `columns` parameter
    .kxi.metaReturn`type`description!(98h;"Return first 100 rows."), / Describe the return of the UDA
    .kxi.metaMisc[enlist[`safe]!enlist 1b]; / States that this UDA is safe to be retried in the event of an error


.kxi.registerUDA `name`query`metadata!(`.example.testUDA`.example.queryFn;metadata)

Refer to Adding logging to UDAs for more details.