Equity Analytics Framework - Adding Custom Analytics

The Equity Analytics Framework calculates a wide number of Order Analytics. This framework is built with flexibility in mind and can easily be extended with custom analytics.

This page provides an example outlining how to add custom analytics with varying levels of complexity.

Note

To achieve the best performance for your custom analytics, use the Order Analytics Utility Functions. These utility functions cover a wide range of scenarios, simplify the development process, and ensure optimal performance for your custom analytics.

Step 1 - Add a patch to the `${PKG_NAME}` Accelerator package

Patches can be used to customize your accelerator package. Refer to the packaging documentation for more information.

The Equity Analytics Framework generates order analytics using data from the Order, Trade and Quote tables. The result is then upserted to the OrderAnalytics table.

If you wish to add custom analytics, you must first ensure that these analytics exist as columns in the OrderAnalytics table.

You must use the kxi CLI to add a patch to your package.

The following example adds a patch called CustomAnalytics.

# Unpack the accelerator package
kxi package unpack ${PKG_NAME}-${VERSION}.kxi

# Create patch to add columns to the OrderAnalytics schema
kxi package add --to ${PKG_NAME} patch --name CustomAnalytics

This generates a YAML patch file in the path ${PKG_NAME}/patches/CustomAnalytics.yaml. We can use this patch file to add custom analytic columns to the OrderAnalytics table.

Update the file ${PKG_NAME}/patches/CustomAnalytics.yaml using your editor of choice.

Note

The CustomAnalytics.yaml file created contains these lines allowing the package structure to be updated:

  databases: []
  pipelines: []
  router: null
  views: []
  deployment_config: null
  data_files: {}

For this example, you should delete these lines from the CustomAnalytics.yaml file.

The yaml snippet below contains all the details we need for this example:

kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:  
  manifest: {}
  tables:
    schemas:
    - name: OrderAnalytics
      columns:   
      - name: reversionAskPrice_30
        type: float
      - name: reversionBidPrice_30
        type: float        
      - name: strikeToCompletionBidMidPrice
        type: float
      - name: strikeToCompletionAskMidPrice
        type: float
      - name: countTradesUnderLimitPrice
        type: int
      - name: sumVolumeUnderLimitPrice
        type: long                   
      - name: myArrivalTradePrice
        type: float        
      - name: myArrivalTradePrice_5
        type: float        
      - name: myArrivalTradePrice_10
        type: float

Here, we are adding several columns to the OrderAnalytics schema, defining the name and type of each column added.

After we add the required columns to our patch, we overlay the patch on our accelerator package using the kxi CLI, as shown below.

# Apply the overlay
kxi package overlay ${PKG_NAME} ${PKG_NAME}/patches/CustomAnalytics.yaml

Step 2 - Define custom analytics

Custom Analytics can be added to the Equity Analytics Framework using a q file that matches the naming convention *eqeaCustomAnalytics.q.

This example creates a *eqeaCustomAnalytics.q file called example.eqeaCustomAnalytics.q.

The *eqeaCustomAnalytics.q file should contain:

A configuration table named .eqea.config.custom.analytics which defines all custom analytics.
Definitions of any custom functions which are used to calculate the custom analytics.

The .eqea.config.custom.analytics table must have the following structure:

Column name	Column type	Description	Example value
analytic	symbol	Name of the analytic. This should correspond to a column defined in `fsi-app-eqea/patches/CustomAnalytics.yaml`	`myFirstAnalytic
analyticType	symbol	A symbol used to filter the config table.	`myFirstAnalyticType`
funcName	symbol	The name of the function used to calculate the analytic.	`myFirstAnalyticFunction`
aggClause	null type	A q parse tree defining the calculation. This can reference columns in the OrderAnalytics, Trade and Quote tables. Although if columns are used from Trade or Quote this should be flagged in the below marketDataTabName column.	`(max;bidPrice)`
marketDataTabName	symbol	The name of the market data tables used by the function. Can be atomic or list of table names.	`Trade` `Quote`
joinTimeOffset	time	Time offset value used to define time to be used for market data join.	00:00:10

Note

All the analytics defined in .eqea.config.custom.analytics are upserted to an internal table named .eqea.analytics.cfg.

All the data from .eqea.config.custom.analytics gets appended to .eqea.analytics.cfg on initialization.#It is recommended to add custom definitions to .eqea.config.custom.analytics instead of directly adding to .eqea.analytics.cfg. This ensures a clear delineation of custom analytics and out-of-the-box analytics. It also ensures a smoother upgrade process when updating to future accelerator versions, ensuring no custom analytics are overwritten.

For further information on .eqea.analytics.cfg, refer to the configuration settings.

In this example, we are defining several custom analytics of varying complexity.

Add the snippet below to our example.eqeaCustomAnalytics.q file:

// Example Custom Analytics
.eqea.config.custom.analytics:flip `analytic`analyticType`funcName`aggClause`marketDataTabName`joinTimeOffset! flip (

    // Examples of adding analytics to the pre-packaged analytic types.
    (`reversionAskPrice_30          ; `reversion        ; `.eqea.analytics.reversion                ; `askPrice                               ; `Quote   ; 00:00:30);    
    (`reversionBidPrice_30          ; `reversion        ; `.eqea.analytics.reversion                ; `bidPrice                               ; `Quote   ; 00:00:30);    

    // Examples of adding simple custom analytic types (simple aggregation on already calculated values in our OrderAnalytics table)
    (`strikeToCompletionBidMidPrice ;`simpleCustomAgg   ;`.example.orderExecution.simpleCustomAgg   ; (%;(+;`arrivalAskPrice;`endAskPrice);2) ; `       ; 0Nt );
    (`strikeToCompletionAskMidPrice ;`simpleCustomAgg   ;`.example.orderExecution.simpleCustomAgg   ; (%;(+;`arrivalAskPrice;`endAskPrice);2) ; `       ; 0Nt );

    // Examples of adding more complex custom analytics (aggregations on tick data with multiple where clauses)
    (`countTradesUnderLimitPrice             ;`tickDataAgg  ;`.example.orderExecution.tickDataAgg  ; (count;`i)                              ; `Trade   ; 0Nt );
    (`sumVolumeUnderLimitPrice               ;`tickDataAgg  ;`.example.orderExecution.tickDataAgg  ; (sum;`volume)                           ; `Trade   ; 0Nt );

    // Examples which use as of joins
    (`myArrivalTradePrice           ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:00 );
    (`myArrivalTradePrice_5         ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:05 );
    (`myArrivalTradePrice_10        ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:10 )

    );

From above we can see various analytics grouped - for the sake of this example - into four analytic types:

reversion
- This is an extension of an existing analytic type that makes use of an existing function .eqea.analytics.reversion.
- Here, you are extending .eqea.analytics.reversion to get the bid and ask price at orderCompletedTime + 30 seconds.
- As this is an extension of an existing function, you do not need to add any custom functions for these analytics.

simpleCustomAgg

This is an example of analytics defined by standard operations on existing columns / analytics in the OrderAnalytics table.
You have specified that a custom function .example.orderExecution.simpleCustomAgg is used to calculate this analytic.
Define .example.orderExecution.simpleCustomAgg in your example.eqeaCustomAnalytics.q file using the below q code:

// Function to be used to generate strikeToCompletionBidMidPrice & strikeToCompletionAskMidPrice
.example.orderExecution.simpleCustomAgg:{[OrderAnalyticsRes; args]

    // Get config for this function
    cfg:select from .eqea.config.custom.analytics where analyticType=`simpleCustomAgg;

    // Define argument dictionary for .eqea.util.simpleAnalytics
    argDict:(`OrderAnalyticsRes`cfg)!(OrderAnalyticsRes;cfg);

    // Calculate analytics using .eqea.util.simpleAnalytics
    OrderAnalyticsRes:.eqea.util.simpleAnalytics[argDict];

    // Return results.
    OrderAnalyticsRes      

    };

tickDataAgg

This is an example of using market data in analytic calculations.
You have specified that a custom function .example.orderExecution.tickDataAgg should be used to calculate this analytic.
Define .example.orderExecution.tickDataAgg in your example.eqeaCustomAnalytics.q file using the below q code:

// Function to be used to calculate countTradesUnderLimitPrice & sumVolumeUnderLimitPrice
.example.orderExecution.tickDataAgg:{[OrderAnalyticsRes; args]

    // Get config required for this function.
    cfg:select from .eqea.config.custom.analytics where analyticType=`tickDataAgg;

    // Our where clause changes depending on whether the order is a buy or a sell.
    // By generating wcList from our input OrderAnalytics table we also ensure the limitPrice value relates to the Order we are analyzing.
    wcList:exec whereClause from select whereClause:(((((`BUY`SELL)!(<=;>=)) orderSideCode),\:(`price)),'limitPrice) from OrderAnalyticsRes;

    // Define argument dictionary for .eqea.util.tickData.getDataAndAggFromCfg
    argDict:`OrderAnalyticsRes`cfg`wcList`startTime`endTime`useQueryWindowAdjustmentFactor`apiargs!(OrderAnalyticsRes;cfg;wcList;`strikeTime;`orderCompletedTime;0b;args);
    OrderAnalyticsRes:.eqea.util.tickData.getDataAndAggFromCfg[argDict];

    // Return results
    OrderAnalyticsRes

    };

aj Example

This is an example of analytics using the 'as of join' functionality.
You have specified that a custom function .example.orderExecution.ajExample should be used to calculate this analytic.

Define .example.orderExecution.ajExample in your example.eqeaCustomAnalytics.q file using the below q code:

// Function to be used to calculate myArrivalTradePrice, myArrivalTradePrice_5 & myArrivalTradePrice_10
.example.orderExecution.ajExample:{[OrderAnalyticsRes; args]

    // Get config required for this function.
    cfg:select from .eqea.config.custom.analytics where analyticType=`ajExample;

    // Define argument dictionary for .eqea.util.asof.ajFromCfg
    argDict:(`OrderAnalyticsRes`cfg`joinOrderTimeCol`kCols`wc`apiargs)!(OrderAnalyticsRes;cfg;`strikeTime;keys OrderAnalyticsRes;();args);

    // Use utility function .eqea.util.asof.ajFromCfg to run analytics that require as-of join functionality.
    OrderAnalyticsRes:.eqea.util.asof.ajFromCfg[argDict];
    OrderAnalyticsRes

    };

Note

For custom functions to work correctly with the framework, they must:

Take table as their only argument.
Return a table which includes all data from the input table with the results of the custom analytics joined to it.

Step 3 - Deploy package with custom analytics

When we are ready to test our custom analytics, do the following:

Add our *eqeaCustomAnalytics.q file to our package.

# Copy our q file containing our analytics to the `src` directory in our Accelerator package
cp example.eqeaCustomAnalytics.q ${PKG_NAME}/src/

First ensure original package has been torn down.
```
kxi pm teardown ${PKG_NAME}
```
Push and deploy your updated package directory using the kxi CLI.
```
kxi pm push ./${PKG_NAME} --force --deploy
```

Troubleshooting

Analytic defined in `.eqea.config.custom.analytics` but not in the `OrderAnalytics` table

If you see the following error message:

"Error running .fsi.eqea.generateOrderAnalytics Analytic Defined in .eqea.config.custom.analytics but not in OrderAnalytics table: someAnalytic, someOtherAnalytic "

This error occurs in the Analytic API if an analytic has been defined in the .eqea.config.custom.analytics but is not present in the OrderAnalytics table.
The missing analytic(s) are called out by the error message. In this example, they are someAnalytic and someOtherAnalytic.
Resolve by ensuring all analytics in .eqea.config.custom.analytics are also defined in your patch file, as shown by the file ${PKG_NAME}/patches/CustomAnalytics.yaml in step 1.
Reapply the patch, repackage and redeploy when you are confident you have aligned the analytics defined in .eqea.config.custom.analytics and your patch file.

Custom function failing

If you see the following error message:

"Error Encountered Running Order Execution Analytic Function: [ .my.custom.function ] - Error Message: [ rank ] "

This error occurs when a function encounters an error.
In this case the function .my.custom.function has encountered a rank error.
To fix this issue debug your code, repackage and redeploy using the steps in the above example.

For more information, refer to debugging custom analytics.

Next steps

Now that your custom analytics have been deployed, test them to ensure they work as intended.

Equity Analytics Framework - Adding Custom Analytics

Step 1 - Add a patch to the ${PKG_NAME} Accelerator package

Step 2 - Define custom analytics

Step 3 - Deploy package with custom analytics

Troubleshooting

Analytic defined in .eqea.config.custom.analytics but not in the OrderAnalytics table

Custom function failing

Next steps

Further reading

Step 1 - Add a patch to the `${PKG_NAME}` Accelerator package

Analytic defined in `.eqea.config.custom.analytics` but not in the `OrderAnalytics` table