Equity Analytics Framework - Adding Custom Analytics

The Equity Analytics Framework calculates a wide number of Order Analytics. This framework is built with flexibility in mind and can easily be extended with custom analytics.

This page provides worked a example outlining how to add custom analytics with varying levels of complexity.

Step 1 - Add a Patch to the fsi-app-eqea Accelerator Package

Patches can be used to customize your accelerator package. Read the customizing the FSI Accelerator documentation.

The Equity Analytics Framework generates order analytics using data from the Order, Trade and Quote tables. The result is then upserted to the OrderAnalytics table. If you wish to add custom analytics you must first ensure that these analytics exist as columns in the OrderAnalytics table. You must use the kxi CLI to add a patch to your package.

The following example uses add a patch called CustomAnalytics.

# Unpack the accelerator package
kxi package unpack fsi-app-eqea-1.0.0.kxi

# Create patch to add columns to the OrderAnalytics schema
kxi package add --to fsi-app-eqea patch --name CustomAnalytics

This generates a YAML patch file in the path fsi-app-eqea/patches/CustomAnalytics.yaml. We can use this patch file to add custom analytic columns to the OrderAnalytics table. Update the file fsi-app-eqea/patches/CustomAnalytics.yaml using your editor of choice.

The below yaml snippet contains all the details we need for this example:

kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:
  uuid: 202c67c8-78da-4ffe-b50f-02a686a29aa0
  manifest: {}
  tables:
    uuid: 458c3288-bf86-4422-a34a-c46eec2c254d
    schemas:
    - name: OrderAnalytics
      columns:   
      - name: reversionAskPrice_30
        type: float
      - name: reversionBidPrice_30
        type: float        
      - name: strikeToCompletionBidMidPrice
        type: float
      - name: strikeToCompletionAskMidPrice
        type: float
      - name: countPriceUnderLimitPrice
        type: int
      - name: sumVolumeUnderLimitPrice
        type: long                   
      - name: myArrivalTradePrice
        type: float        
      - name: myArrivalTradePrice_5
        type: float        
      - name: myArrivalTradePrice_10
        type: float                
  databases: []
  pipelines: []
  router: null
  views: []
  deployment_config: null
  data_files: {}

Here we can see we are adding several columns to the OrderAnalytics schema, defining the name and type of each column added.

After we add the required columns to our patch, we overlay the patch on our accelerator package using the kxi CLI.

# Apply the overlay
kxi package overlay fsi-app-eqea fsi-app-eqea/patches/CustomAnalytics.yaml

# TODO: PACKAGING BUG - REMOVE THIS AFTER RESOLUTION!!!
rm -rf fsi-app-eqea/databases/fsi-core-db/tables

Step 2 - Define Custom Analytics

Custom Analytics can be added to the Equity Analytics Framework using a q file that matches the naming convention *eqeaCustomAnalytics*.q.

This example creates a *eqeaCustomAnalytics*.q file called example.eqeaCustomAnalytics.q.

The *eqeaCustomAnalytics*.q file should contain:

A configuration table named .eqea.analytics.cfg which defines all custom analytics.
Definitions of any custom functions which are used to calculate the custom analytics.

The .eqea.analytics.cfg table must have the following structure:

Column name	Column type	Description	Example value
analytic	symbol	Name of the analytic. This should correspond to a column defined in `fsi-app-eqea/patches/CustomAnalytics.yaml`	`myFirstAnalytic
analyticType	symbol	A symbol used to filter the config table.	`myFirstAnalyticType
funcName	symbol	The name of the function used to calculate the analytic.	`myFirstAnalyticFunction
aggClause	null type	A q parse tree defining the calculation. This can reference columns in the OrderAnalytics, Trade and Quote tables. Although if columns are used from Trade or Quote this should be flagged in the below marketDataTabName column.	(max;`bidPrice)
marketDataTabName	symbol	The name of the market data tables used by the function. Can be atomic or list of table names.	`Trade`Quote
joinTimeOffset	time	Time offset value used to define time to be used for market data join.	00:00:10

In this example, we are defining several custom analytics of varying complexity. Add the below snippet to our example.eqeaCustomAnalytics.q file:

// Example Custom Analytics
.eqea.config.custom.analytics:flip `analytic`analyticType`funcName`aggClause`marketDataTabName`joinTimeOffset! flip (

    // Examples of adding analytics to the pre-packaged analytic types.
    (`reversionAskPrice_30          ; `reversion        ; `.eqea.analytics.reversion                ; `askPrice                               ; `Quote   ; 00:00:30);    
    (`reversionBidPrice_30          ; `reversion        ; `.eqea.analytics.reversion                ; `bidPrice                               ; `Quote   ; 00:00:30);    

    // Examples of adding simple custom analytic types (simple aggregation on already calculated values in our OrderAnalytics table)
    (`strikeToCompletionBidMidPrice ;`simpleCustomAgg   ;`.example.orderExecution.simpleCustomAgg   ; (%;(+;`arrivalAskPrice;`endAskPrice);2) ; `       ; 0Nt );
    (`strikeToCompletionAskMidPrice ;`simpleCustomAgg   ;`.example.orderExecution.simpleCustomAgg   ; (%;(+;`arrivalAskPrice;`endAskPrice);2) ; `       ; 0Nt );

    // Examples of adding more complex custom analytics (aggregations on tick data with multiple where clauses)
    (`countPriceUnderLimitPrice              ;`tickDataAgg  ;`.example.orderExecution.tickDataAgg  ; (count;`i)                              ; `Trade   ; 0Nt );
    (`sumVolumeUnderLimitPrice               ;`tickDataAgg  ;`.example.orderExecution.tickDataAgg  ; (sum;`volume)                           ; `Trade   ; 0Nt );

    // Examples which use as of joins
    (`myArrivalTradePrice           ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:00 );
    (`myArrivalTradePrice_5         ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:05 );
    (`myArrivalTradePrice_10        ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:10 )

    );

From above we can see various analytics grouped - for the sake of this example - into four analytic types:

reversion
- This is an extension of an existing analytic type.
- Here, we are adding an extra interval for our reversion analytics.
- We do not need to add any custom functions to generate this analytic.
simpleCustomAgg
- This is an example of a simple calculations that generate an analytics based on to columns that already exist in the OrderAnalytics table.
- We have specified that a custom function .example.orderExecution.simpleCustomAgg will be used to calculate this analytic.
- For this to work we must define .example.orderExecution.simpleCustomAgg in our example.eqeaCustomAnalytics.q file using the below q code:

```q // Function to be used to generate strikeToCompletionBidMidPrice & strikeToCompletionAskMidPrice .example.orderExecution.simpleCustomAgg:{[OrderAnalyticsRes]

  // We can use a util function `.eqea.util.runSimpleAnalytic` to run analytics that are dependent on columns that already exist in the OrderAnalytics table
 cfg:select from .eqea.analytics.cfg where analyticType=`simpleCustomAgg;
 .eqea.util.simpleAnalytics[OrderAnalyticsRes;cfg]

 };

tickDataAgg
- This is an example of using market data in analytic calculations.
- We have specified that a custom function .example.orderExecution.tickDataAgg should be used to calculate this analytic.
- To be successful, we must define .example.orderExecution.tickDataAgg in our example.eqeaCustomAnalytics.q file using the below q code:
```q .example.orderExecution.tickDataAgg:{[OrderAnalyticsRes]

// We can use .eqea.util.runTickDataAnalytics to run analytics on tick data tables cfg:select from .eqea.analytics.cfg where analyticType=tickDataAgg; // Our where clause changes depending on whether the order is a buy or a sell. // By generating wcList from our input OrderAnalytics table we also ensure the limitPrice value relates to the Order we are analyzing. wcList:exec whereClause from select whereClause:(((((BUYSELL)!(<=;>=)) orderSideCode),\:(price)),'limitPrice) from OrderAnalyticsRes; OrderAnalyticsRes:.eqea.util.tickData.getDataAndAggFromCfg[OrderAnalyticsRes;cfg;wcList;strikeTime;orderCompletedTime]; OrderAnalyticsRes

};
aj Example
- This is an example of analytics using the 'as of join' functionality.
- We have specified that a custom function .example.orderExecution.ajExample should be used to calculate this analytic.
- To be successful, we must define .example.orderExecution.ajExample in our example.eqeaCustomAnalytics.q file using the below q code:

.example.orderExecution.ajExample:{[OrderAnalyticsRes]

    // We can use `.eqea.util.runTickDataAnalytics` to run analytics which retrieve values from tick data using an as of joins
    cfg:select from .eqea.analytics.cfg where analyticType=`ajExample;
    OrderAnalyticsRes:.eqea.util.asof.ajFromCfg[OrderAnalyticsRes;cfg;`strikeTime];
    OrderAnalyticsRes

    };

!!! "note:

* For custom functions to work correctly with the framework, they **must:**
  * Take table as its only argument.
  * Return a table which includes **all data** from the input table **with the results of the custom analytics joined to it**."

## Step 3 - Deploy Our Package with Custom Analytics

When we are ready to test our custom analytics, do the following:

- add our `*eqeaCustomAnalytics*.q` file to our package.

```shell
# Copy our q file containing our analytics to the `src` directory in our Accelerator package
cp example.eqeaCustomAnalytics.q fsi-app-eqea/src/

Re-package using the kxi CLI.
It's recommended that the package is renamed during repackaging, so that it can be distinguished as a custom package.
In this example it's renamed to fsi-app-custom-eqea.

!!! "note:

When renaming custom accelerator packages, the new name must have the prefix fsi-app-

# Repackage with new name 
kxi package packit fsi-app-eqea --tag --package-name fsi-app-custom-eqea
``` "

- Redeploy using the kxi CLI.

```shell
kxi package push /tmp/artifacts/fsi-app-custom-eqea-1.0.0.kxi
kxi package deploy fsi-app-custom-eqea-1.0.0.kxi

Troubleshooting

Analytic defined in `.eqea.analytics.cfg` but not in the `OrderAnalytics` table

If you see the following error message:

"Error running .fsi.eqea.generateOrderAnalytics Analytic Defined in .eqea.analytics.cfg but not in OrderAnalytics table: someAnalytic, someOtherAnalytic "

This error occurs in the Analytic API if an analytic has been defined in the .eqea.analytics.cfg but is not present in the OrderAnalytics table.
The missing analytic(s) are called out by the error message. In this example, they are someAnalytic and someOtherAnalytic.
Resolve by ensuring all analytics in .eqea.analytics.cfg are also defined in your patch file. (As shown by the file fsi-app-eqea/patches/CustomAnalytics.yaml in step 1.)
Reapply the patch, repackage and redeploy when you are confident you have aligned the analytics defined in .eqea.analytics.cfg and your patch file.

Custom Function Failing