Creating User Defined Analytics (UDAs)

Introduction

User Defined Analytics (UDAs) enable you to define new APIs that are callable through the Service Gateway (SG). UDAs augment the standard set of APIs available in the kdb Insights system with application logic specific to your business needs.

A UDA consists of a query function that reads data from a Data Access Process and optionally an aggregation function that combines the partials from the query function to produce the final result, which is then returned. These functions can be loaded into the Data Access Processes (DAPs) and the Aggregators in kdb Insights using packages.

Steps to create a User Defined Analytic (UDA)

This section provides an example UDA to guide you through the steps required to create the code necessary to make it callable using the kdb Insights Service Gateway, once it is added to a package and deployed.

The .exampleuda.countBy UDA counts the number of records per unique values of specified columns for a chosen table and time range.

.exampleuda.countBy code

// Custom count by UDA.

// Define query function that selects specified columns for aggregation.
.exampleuda.countByQuery:{[table;byCols;startTS;endTS;byCols]
    byCols,:();
    data:.kxi.selectTable`table`startTS`endTS`agg!(table;startTS;endTS;byCols!byCols);
    .kxi.response.ok(byCols;data)
    }

// Define aggregation function that counts number of entries by specified columns.
.exampleuda.countByAgg:{[tbls]
    t:raze last each tbls;
    bc:first first tbls;
    res:?[t;();bc!bc;enlist[`cnt]!enlist(count;`i)];
    .kxi.response.ok res
    }

// Define metadata.
metadata:.kxi.metaDescription["Custom UDA - does a count by."],
    .kxi.metaMisc[enlist[`safe]!enlist 1b],
    .kxi.metaParam[`name`type`isReq`description!(`table;-11h;1b;"Table name.")],
    .kxi.metaParam[`name`type`isReq`description!(`byCols;11 -11h;1b;"Column(s) to count by.")],
    .kxi.metaParam[`name`type`isReq`description!(`startTS;-12h;1b;"Start time (inclusive).")],
    .kxi.metaParam[`name`type`isReq`description!(`endTS;-12h;1b;"End time (exclusive).")],
    .kxi.metaReturn`type`description!(98h;"Count by specified columns.");

// Registration.
.kxi.registerUDA `name`query`aggregation`metadata!(`.exampleuda.countBy;`.exampleuda.countByQuery;`.exampleuda.countByAgg;metadata);

It is recommended to keep the function definitions and registration in the same file to ensure consistent definitions across DAPs and Aggregators. For additional information, refer to the UDA examples.

1. Define the Query Function

The query function reads data from a Data Access Process and returns a table of data.

1. Create a file and add the following code:

   // Define query function that selects specified columns for aggregation.
   .exampleuda.countByQuery:{[table;byCols;startTS;endTS;byCols]
       byCols,:();
       data:.kxi.selectTable`table`startTS`endTS`agg!(table;startTS;endTS;byCols!byCols);
       .kxi.response.ok(byCols;data)
       }
   

It is highly recommended that you use the helper functions when retrieving data from the DAPs.

The countByQuery function utilizes the .kxi.selectTable helper function to collect the required data from the specified table and time range.

In the example the arguments are a list of values. You can define the arguments as either:

• A list of values.

  For example:

  .example.queryFn:{[table;startTS;endTS;columns]
      columns:$[-11h = type columns;enlist columns;columns];
      data:.kxi.selectTable`table`startTS`endTS`agg!(table;startTS;endTS;columns!columns); // Retrieve data within specified time range
      .kxi.response.ok data
      };
  

• A dictionary of keys named args.

  Warning

  You must name the dictionary of keys args. Any other name causes the DAP to treat the function as a one parameter function, which ignores the auto-casting of REST parameters.

  For example:

  .example.queryFn:{[args]
      columns:$[-11h = type columns:args`columns;enlist columns;columns];
      filter:enlist (<;`i;100); // Note for partitioned tables, will return first 100 per date
      .kxi.response.ok ?[args`table;filter;0b;columns!columns]
      };
  

Note

You must wrap the response with the helper function .kxi.response.ok to indicate successful execution. For examples of the required response shape, refer to the generating a response header section.

If an aggregation function is not defined, the query function must return a table for the default operator, kdb+ raze to successfully combine the partials.

2. Define the Aggregation Function

The aggregation function customizes aggregation based on the specific needs of the UDA. It combines a list of results, referred to as partials, obtained by executing the query function on each DAP.

1. Add the aggregation function to the file created in the previous step:

   // Define aggregation function that counts number of entries by specified columns.
   .exampleuda.countByAgg:{[tbls]
       t:raze last each tbls;
       bc:first first tbls;
       res:?[t;();bc!bc;enlist[`cnt]!enlist(count;`i)];
       .kxi.response.ok res
       }
   

The startTS and endTS parameters of the query function direct the query to specific tiers. The .exampleuda.countByAgg function receives a table from each tier and aggregates the counts per column across the tables returned by countByQuery.

If no aggregation function is defined, the kdb+ raze operator is used.

• The query function must return a table for the raze operator to successfully combine the partials.

• With memory mapped table types such as basic or splayed, ensure that the query function includes table as a parameter. This ensures correct routing to a single DAP for data requests and prevents duplicated results in the response. For more information, refer to the aggregation examples.

3. Add Metadata

Define metadata for the UDA to describe its purpose, parameters, and return values:

1. Add the metadata to the file:

// Define metadata.
metadata:.kxi.metaDescription["Custom UDA - does a count by."],
    .kxi.metaMisc[enlist[`safe]!enlist 1b],
    .kxi.metaParam[`name`type`isReq`description!(`table;-11h;1b;"Table name.")],
    .kxi.metaParam[`name`type`isReq`description!(`byCols;11 -11h;1b;"Column(s) to count by.")],
    .kxi.metaParam[`name`type`isReq`description!(`startTS;-12h;1b;"Start time (inclusive).")],
    .kxi.metaParam[`name`type`isReq`description!(`endTS;-12h;1b;"End time (exclusive).")],
    .kxi.metaReturn`type`description!(98h;"Count by specified columns.");

Meta-Building APIs

The .kxi namespace includes a set of meta-building APIs for defining metadata. The following APIs simplify the process of defining metadata entries for UDAs. For further examples of how to use these APIs, refer to the Examples.

• .kxi.metaDescription - Creates a description entry for a UDA's metadata.

  • Parameters:

    • descr: string - The description of the UDA.

• .kxi.metaParam - Creates a parameter entry for a UDA's metadata. These parameters should match the inputs of the query function.

  • Parameters:

    • param: dictionary - Dictionary containing any subset of the following keys:
    • name: symbol - The name of the parameter.
    • type: short[]|short - The possible types for the parameter.

    Info

    In the case of a REST request, this entry helps automatically cast the input to the correct type. For example, symbols are processed as strings, but specifying the type as -11h prompts kdb Insights to cast it. If multiple types are specified, auto-casting uses the first type in the list.

    • isReq: Boolean - Indicates whether the parameter is required.
    • default: any - The default value of the parameter if it is not required.
    • description: string - Plain text description of the parameter.

• .kxi.metaReturn - Creates a return entry for a UDA's metadata. This entry should match the return type of the aggregation function.

  • Parameters:
    • return: dictionary - Dictionary containing any subset of the following keys:
    • type: short|short[] - The possible types for the return value.
    • description: string - Plain text description of the return value.

• .kxi.metaMisc - Creates a miscellaneous metadata entry.

  • Parameters:
    • misc: dictionary - Dictionary containing any subset of the supported miscellaneous fields.
    • safe: Boolean - Indicates whether the UDA can be safely retried in the event of a failure.

4. Register the UDA

Register the UDA with the appropriate names for query and aggregation functions, and provide the metadata to ensure the UDA can be called from the Service Gateway. This registration is done using the .kxi.registerUDA function.

1. Add the registration to the file:

   .kxi.registerUDA `name`query`aggregation`metadata!(`.exampleuda.countBy;`.exampleuda.countByQuery;`.exampleuda.countByAgg;metadata);
   

.kxi.registerUDA Parameters

The .kxi.registerUDA function accepts a dictionary with the following keys:

Key Name	Required	Type	Description
name	Yes	Symbol	The name of the UDA when called using the Service Gateway. *
query	Yes	Symbol	The name of the function that runs on the DAPs and retrieves the partials for aggregation.
aggregation	No	Symbol	The name of the function that runs on the aggregator and combines the partial results from the DAPs.
metadata	No	List	Metadata about the properties of the UDA. It's recommended to use the metadata builders described in the Meta-building APIs section.

Next steps

• To add this UDA to a package, follow the Adding a UDA to a package guide.

• Refer to UDA examples for more examples of User Defined Analytics.