Equity Analytics Framework - Adding Custom Analytics
The Equity Analytics Framework calculates a wide number of Order Analytics. This framework is built with flexibility in mind and can easily be extended with custom analytics.
This page provides worked a example outlining how to add custom analytics with varying levels of complexity.
Step 1 - Add a Patch to the fsi-app-eqea Accelerator Package
Patches can be used to customize your accelerator package. Read the customizing the FSI Accelerator documentation.
The Equity Analytics Framework generates order analytics using data from the Order, Trade and Quote tables.
The result is then upserted to the OrderAnalytics
table.
If you wish to add custom analytics you must first ensure that these analytics exist as columns in the OrderAnalytics
table.
You must use the kxi CLI to add a patch to your package.
The following example uses add a patch called CustomAnalytics
.
# Unpack the accelerator package
kxi package unpack fsi-app-eqea-1.0.0.kxi
# Create patch to add columns to the OrderAnalytics schema
kxi package add --to fsi-app-eqea patch --name CustomAnalytics
This generates a YAML patch file in the path fsi-app-eqea/patches/CustomAnalytics.yaml
.
We can use this patch file to add custom analytic columns to the OrderAnalytics
table.
Update the file fsi-app-eqea/patches/CustomAnalytics.yaml
using your editor of choice.
The below yaml snippet contains all the details we need for this example:
kind: Package
apiVersion: pakx/v1
metadata:
name: target
spec:
uuid: 202c67c8-78da-4ffe-b50f-02a686a29aa0
manifest: {}
tables:
uuid: 458c3288-bf86-4422-a34a-c46eec2c254d
schemas:
- name: OrderAnalytics
columns:
- name: reversionAskPrice_30
type: float
- name: reversionBidPrice_30
type: float
- name: strikeToCompletionBidMidPrice
type: float
- name: strikeToCompletionAskMidPrice
type: float
- name: countPriceUnderLimitPrice
type: int
- name: sumVolumeUnderLimitPrice
type: long
- name: myArrivalTradePrice
type: float
- name: myArrivalTradePrice_5
type: float
- name: myArrivalTradePrice_10
type: float
databases: []
pipelines: []
router: null
views: []
deployment_config: null
data_files: {}
Here we can see we are adding several columns to the OrderAnalytics
schema, defining the name and type of each column added.
After we add the required columns to our patch, we overlay the patch on our accelerator package using the kxi CLI.
# Apply the overlay
kxi package overlay fsi-app-eqea fsi-app-eqea/patches/CustomAnalytics.yaml
# TODO: PACKAGING BUG - REMOVE THIS AFTER RESOLUTION!!!
rm -rf fsi-app-eqea/databases/fsi-core-db/tables
Step 2 - Define Custom Analytics
Custom Analytics can be added to the Equity Analytics Framework using a q file that matches the naming convention *eqeaCustomAnalytics*.q
.
This example creates a *eqeaCustomAnalytics*.q
file called example.eqeaCustomAnalytics.q
.
The *eqeaCustomAnalytics*.q
file should contain:
- A configuration table named
.eqea.analytics.cfg
which defines all custom analytics. - Definitions of any custom functions which are used to calculate the custom analytics.
The .eqea.analytics.cfg
table must have the following structure:
Column name | Column type | Description | Example value |
---|---|---|---|
analytic | symbol | Name of the analytic. This should correspond to a column defined in fsi-app-eqea/patches/CustomAnalytics.yaml |
`myFirstAnalytic |
analyticType | symbol | A symbol used to filter the config table. | `myFirstAnalyticType |
funcName | symbol | The name of the function used to calculate the analytic. | `myFirstAnalyticFunction |
aggClause | null type | A q parse tree defining the calculation. This can reference columns in the OrderAnalytics, Trade and Quote tables. Although if columns are used from Trade or Quote this should be flagged in the below marketDataTabName column. | (max;`bidPrice) |
marketDataTabName | symbol | The name of the market data tables used by the function. Can be atomic or list of table names. | `Trade`Quote |
joinTimeOffset | time | Time offset value used to define time to be used for market data join. | 00:00:10 |
In this example, we are defining several custom analytics of varying complexity. Add the below snippet to our example.eqeaCustomAnalytics.q
file:
// Example Custom Analytics
.eqea.config.custom.analytics:flip `analytic`analyticType`funcName`aggClause`marketDataTabName`joinTimeOffset! flip (
// Examples of adding analytics to the pre-packaged analytic types.
(`reversionAskPrice_30 ; `reversion ; `.eqea.analytics.reversion ; `askPrice ; `Quote ; 00:00:30);
(`reversionBidPrice_30 ; `reversion ; `.eqea.analytics.reversion ; `bidPrice ; `Quote ; 00:00:30);
// Examples of adding simple custom analytic types (simple aggregation on already calculated values in our OrderAnalytics table)
(`strikeToCompletionBidMidPrice ;`simpleCustomAgg ;`.example.orderExecution.simpleCustomAgg ; (%;(+;`arrivalAskPrice;`endAskPrice);2) ; ` ; 0Nt );
(`strikeToCompletionAskMidPrice ;`simpleCustomAgg ;`.example.orderExecution.simpleCustomAgg ; (%;(+;`arrivalAskPrice;`endAskPrice);2) ; ` ; 0Nt );
// Examples of adding more complex custom analytics (aggregations on tick data with multiple where clauses)
(`countPriceUnderLimitPrice ;`tickDataAgg ;`.example.orderExecution.tickDataAgg ; (count;`i) ; `Trade ; 0Nt );
(`sumVolumeUnderLimitPrice ;`tickDataAgg ;`.example.orderExecution.tickDataAgg ; (sum;`volume) ; `Trade ; 0Nt );
// Examples which use as of joins
(`myArrivalTradePrice ;`ajExample ;`.example.orderExecution.ajExample ; `price ; `Trade ; 00:00:00 );
(`myArrivalTradePrice_5 ;`ajExample ;`.example.orderExecution.ajExample ; `price ; `Trade ; 00:00:05 );
(`myArrivalTradePrice_10 ;`ajExample ;`.example.orderExecution.ajExample ; `price ; `Trade ; 00:00:10 )
);
From above we can see various analytics grouped - for the sake of this example - into four analytic types:
-
reversion
- This is an extension of an existing analytic type.
- Here, we are adding an extra interval for our reversion analytics.
- We do not need to add any custom functions to generate this analytic.
-
simpleCustomAgg
- This is an example of a simple calculations that generate an analytics based on to columns that already exist in the OrderAnalytics table.
- We have specified that a custom function
.example.orderExecution.simpleCustomAgg
will be used to calculate this analytic. - For this to work we must define
.example.orderExecution.simpleCustomAgg
in ourexample.eqeaCustomAnalytics.q
file using the below q code:
```q // Function to be used to generate strikeToCompletionBidMidPrice & strikeToCompletionAskMidPrice .example.orderExecution.simpleCustomAgg:{[OrderAnalyticsRes]
// We can use a util function `.eqea.util.runSimpleAnalytic` to run analytics that are dependent on columns that already exist in the OrderAnalytics table
cfg:select from .eqea.analytics.cfg where analyticType=`simpleCustomAgg;
.eqea.util.simpleAnalytics[OrderAnalyticsRes;cfg]
};
-
tickDataAgg
- This is an example of using market data in analytic calculations.
- We have specified that a custom function
.example.orderExecution.tickDataAgg
should be used to calculate this analytic. - To be successful, we must define
.example.orderExecution.tickDataAgg
in ourexample.eqeaCustomAnalytics.q
file using the below q code:
```q .example.orderExecution.tickDataAgg:{[OrderAnalyticsRes]
// We can use
.eqea.util.runTickDataAnalytics
to run analytics on tick data tables cfg:select from .eqea.analytics.cfg where analyticType=tickDataAgg; // Our where clause changes depending on whether the order is a buy or a sell. // By generating wcList from our input OrderAnalytics table we also ensure the limitPrice value relates to the Order we are analyzing. wcList:exec whereClause from select whereClause:(((((
BUYSELL)!(<=;>=)) orderSideCode),\:(
price)),'limitPrice) from OrderAnalyticsRes; OrderAnalyticsRes:.eqea.util.tickData.getDataAndAggFromCfg[OrderAnalyticsRes;cfg;wcList;strikeTime;
orderCompletedTime]; OrderAnalyticsRes};
-
aj Example
- This is an example of analytics using the 'as of join' functionality.
- We have specified that a custom function
.example.orderExecution.ajExample
should be used to calculate this analytic. - To be successful, we must define
.example.orderExecution.ajExample
in ourexample.eqeaCustomAnalytics.q
file using the below q code:
.example.orderExecution.ajExample:{[OrderAnalyticsRes]
// We can use `.eqea.util.runTickDataAnalytics` to run analytics which retrieve values from tick data using an as of joins
cfg:select from .eqea.analytics.cfg where analyticType=`ajExample;
OrderAnalyticsRes:.eqea.util.asof.ajFromCfg[OrderAnalyticsRes;cfg;`strikeTime];
OrderAnalyticsRes
};
!!! "note:
* For custom functions to work correctly with the framework, they **must:**
* Take table as its only argument.
* Return a table which includes **all data** from the input table **with the results of the custom analytics joined to it**."
## Step 3 - Deploy Our Package with Custom Analytics
When we are ready to test our custom analytics, do the following:
- add our `*eqeaCustomAnalytics*.q` file to our package.
```shell
# Copy our q file containing our analytics to the `src` directory in our Accelerator package
cp example.eqeaCustomAnalytics.q fsi-app-eqea/src/
- Re-package using the kxi CLI.
- It's recommended that the package is renamed during repackaging, so that it can be distinguished as a custom package.
- In this example it's renamed to
fsi-app-custom-eqea
.
!!! "note:
- When renaming custom accelerator packages, the new name must have the prefix
fsi-app-
# Repackage with new name
kxi package packit fsi-app-eqea --tag --package-name fsi-app-custom-eqea
``` "
- Redeploy using the kxi CLI.
```shell
kxi package push /tmp/artifacts/fsi-app-custom-eqea-1.0.0.kxi
kxi package deploy fsi-app-custom-eqea-1.0.0.kxi
Troubleshooting
Analytic defined in .eqea.analytics.cfg
but not in the OrderAnalytics
table
If you see the following error message:
"Error running .fsi.eqea.generateOrderAnalytics Analytic Defined in .eqea.analytics.cfg but not in OrderAnalytics table: someAnalytic, someOtherAnalytic "
- This error occurs in the Analytic API if an analytic has been defined in the
.eqea.analytics.cfg
but is not present in theOrderAnalytics
table. - The missing analytic(s) are called out by the error message. In this example, they are
someAnalytic
andsomeOtherAnalytic
. - Resolve by ensuring all analytics in .eqea.analytics.cfg are also defined in your patch file. (As shown by the file
fsi-app-eqea/patches/CustomAnalytics.yaml
in step 1.) - Reapply the patch, repackage and redeploy when you are confident you have aligned the analytics defined in
.eqea.analytics.cfg
and your patch file.
Custom Function Failing
If you see the following error message:
"Error Encountered Running Order Execution Analytic Function: [ .my.custom.function ] - Error Message: [ rank ] "
- This error occurs when a function encounters an error.
- In this case the function
.my.custom.function
has encountered a rank error. - To fix this issue debug your code, repackage and redeploy using the steps in the above example.