Skip to content

Equity Analytics Framework - Util Functions

The Equity Analytics Framework calculates a wide number of Order Analytics. This framework is built with flexibility in mind, and can easily be extended with custom analytics.

A number of utility functions have been written to enable development of custom analytics.

.eqea.util.simpleAnalytics

The Util function enables users to run "simple" analytic, that is, analytics that make use of columns already present in the OrderAnalytics table.

The function .eqea.util.simpleAnalytics runs on a subset of the .eqea.analytics.cfg.

Analytics that make use of .eqea.util.simpleAnalytics should have at a minimum the following values populated in .eqea.analytics.cfg:

  • analytic
  • analyticType
  • funcName
  • aggClause

Parameters:

Name Type Description
OrderAnalyticsRes table Table containing all OrderAnalytics data - contains the results of all analytics preceeding the current analytic type.
cfg table Subset of .eqea.analytics.cfg - contains all details required for current batch of analytics.

Returns:

Type Description
table Table containing all Analytic Results.

Example usage:

Writing example custom function .example.orderExecution.simpleCustomAgg:

  • Populating required values in .eqea.analytics.cfg:
.eqea.analytics.cfg:flip `analytic`analyticType`funcName`aggClause`marketDataTabName`joinTimeOffset! flip (

    // Examples of adding simple custom analytic types (simple aggregation on already calculated values in our OrderAnalytics table)
    (`strikeToCompletionBidMidPrice ;`simpleCustomAgg   ;`.example.orderExecution.simpleCustomAgg   ; (%;(+;`arrivalAskPrice;`endAskPrice);2) ; `       ; 0Nt );
    (`strikeToCompletionAskMidPrice ;`simpleCustomAgg   ;`.example.orderExecution.simpleCustomAgg   ; (%;(+;`arrivalAskPrice;`endAskPrice);2) ; `       ; 0Nt )

    );
  • Contents of example function: .example.orderExecution.simpleCustomAgg
.example.orderExecution.simpleCustomAgg:{[OrderAnalyticsRes]

    // We can use a util function `.eqea.util.runSimpleAnalytic` to run analytics that are dependent on columns that already exist in the OrderAnalytics table
    cfg:select from .eqea.analytics.cfg where analyticType=`simpleCustomAgg;
   .eqea.util.simpleAnalytics[OrderAnalyticsRes;cfg]

    };

.eqea.util.simpleAnalyticsWithWc

Util function to enable users to run "simple" analytics when a where clause is in the cfg i.e. analytics that make use of columns already present in OrderAnalytics table.

The function .eqea.util.simpleAnalyticsWithWc runs on a subset of the .eqea.analytics.prevailing.cfg table.

Analytics that make use of .eqea.util.simpleAnalyticsWithWc should have at a minimum the following values populated in the table passed in to the cfg argument:

  • whereClause
  • aggClause

Parameters:

Name Type Description
OrderAnalyticsRes table Table containing all OrderAnalytics data - contains the results of all analytics preceeding the current analytic type.
cfg table Table containing all details required for current batch of analytics, where clause and aggregation clause to apply

Returns:

Type Description
table Table containing all Analytic Results.

.eqea.util.tickData.getDataAndAggFromCfg

The Util function is used to query tick data and aggregate based on a subset of the .eqea.analytics.cfg table. Analytics that make use of .eqea.util.tickData.getDataAndAggFromCfg should have the minimum following values populated in .eqea.analytics.cfg:

  • analytic
  • analyticType
  • funcName
  • aggClause
  • marketDataTabName

Parameters:

Name Type Description**
OrderAnalyticsRes table Table containing all OrderAnalytics data - contains the results of all analytics preceeding the current analytic type.
cfg table Subset of .eqea.analytics.cfg - contains all details required for current batch of analytics.
wcList list List of where clauses to be used when querying tick data. NOTE: this list should have the same number of items as there are rows in the Order data. NOTE: if no additional where clause logic is needed then simply use ().
startTime symbol | parse tree Relative start time. Can either be an atomic symbol indictating a timestamp column or a parse tree that conforms to the format: (operator;timeColumn;timestamp), e.g. (-;strikeTime;0D00:01:00), (+;orderCompletedTime;0D00:00:30)
endTime symbol | parse tree Relative end time. Can either be an atomic symbol indictating a timestamp column or a parse tree that conforms to the format: (operator;timeColumn;timestamp), e.g. (-;strikeTime;0D00:01:00), (+;orderCompletedTime;0D00:00:30)

Returns:

Type Description
table Keyed table containing all OrderAnalytic results to this point.

Example usage:

Writing example custom function .example.orderExecution.tickDataAgg:

  • Populating required values in .eqea.analytics.cfg:
.eqea.analytics.cfg:flip `analytic`analyticType`funcName`aggClause`marketDataTabName`joinTimeOffset! flip (

    // Examples of adding more complex custom analytics (aggregations on tick data with multiple where clauses)
    (`countPriceUnderLimitPrice              ;`tickDataAgg  ;`.example.orderExecution.tickDataAgg  ; (count;`i)                              ; `Trade   ; 0Nt );
    (`sumVolumeUnderLimitPrice               ;`tickDataAgg  ;`.example.orderExecution.tickDataAgg  ; (sum;`volume)                           ; `Trade   ; 0Nt )

    );
  • Contents of example function: .example.orderExecution.tickDataAgg
.example.orderExecution.tickDataAgg:{[OrderAnalyticsRes]

    // We can use `.eqea.util.runTickDataAnalytics` to run analytics on tick data tables
    cfg:select from .eqea.analytics.cfg where analyticType=`tickDataAgg;
    // Our where clause changes depending on whether the order is a buy or a sell.
    // By generating wcList from our input OrderAnalytics table we also ensure the limitPrice value relates to the Order we are analyzing.
    wcList:exec whereClause from select whereClause:(((((`BUY`SELL)!(<=;>=)) orderSideCode),\:(`price)),'limitPrice) from OrderAnalyticsRes;
    OrderAnalyticsRes:.eqea.util.tickData.getDataAndAggFromCfg[OrderAnalyticsRes;cfg;wcList;`strikeTime;`orderCompletedTime];
    OrderAnalyticsRes

    };

.eqea.util.tickData.getDataAndAggregate

The Util function can be used to query tick data and aggregate based on input arguments.

This is a low-level function, essentially a wrapper for a functional select, to be used for complex use cases that are incompatible with .eqea.util.tickData.getDataAndAggFromCfg.

Parameters:

Name Type Description
tickTabName symbol Name of the tick data table to query.
id symbol InstrumentID to query.
tw timestamp[2] Time period of interest.
wc list[] Where clause for query.
bc dict / boolean By clause for query. NOTE: if no by clause needed then use 0b.
aggDict dict Aggregation / column clause for query.

Returns:

Type Description
table Table with the results of our query.

.eqea.util.asof.ajFromCfg

This function calculates configured analytics that are based on as-of join logic.

This function:

  1. Builds a time rack for each order based on whats configured in the joinTimeOffset column of the .eqea.analytics.cfg table.
  2. Performs an as-of join, joining the time rack to the table specified by the marketDataTabName column of the .eqea.analytics.cfg table.
  3. Pivots the result of the as-of join such that the data is pivoted to a single row per order, under column names as defined by the analytic column of the .eqea.analytics.cfg table.
  4. Returns the result to be joined onto our input OrderAnalytics data.

Analytics that make use of .eqea.util.asof.ajFromCfg should have the minimum following values populated in .eqea.analytics.cfg:

  • analytic
  • analyticType
  • funcName
  • aggClause
  • marketDataTabName
  • joinTimeOffset

Parameters:

Name Type Description
OrderAnalyticsRes table Table containing all OrderAnalytics data - contains the results of all analytics preceding the current analytic type.
cfg table Subset of .eqea.analytics.cfg - contains all details required for current batch of analytics.
joinOrderTimeCol symbol Time column from the Order table. This data what we use as the start point for the time rack table.
wc parse tree Where clause to filter tick data by.

Returns:

Type Description
table Full OrderAnalytics table up to this point.

Example usage:

Writing example custom function .example.orderExecution.ajExample:

  • Populating required values in .eqea.analytics.cfg:
.eqea.analytics.cfg:flip `analytic`analyticType`funcName`aggClause`marketDataTabName`joinTimeOffset! flip (

    // Examples which use as of joins
    (`myArrivalTradePrice           ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:00 );
    (`myArrivalTradePrice_5         ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:05 );
    (`myArrivalTradePrice_10        ;`ajExample   ;`.example.orderExecution.ajExample                ; `price                                ; `Trade     ; 00:00:10 )

    );
  • Contents of example function: .example.orderExecution.ajExample
.example.orderExecution.ajExample:{[OrderAnalyticsRes]

    // We can use `.eqea.util.runTickDataAnalytics` to run analytics which retrieve values from tick data using an as of joins
    cfg:select from .eqea.analytics.cfg where analyticType=`ajExample;
    OrderAnalyticsRes:.eqea.util.asof.ajFromCfg[OrderAnalyticsRes;cfg;`strikeTime;()];
    OrderAnalyticsRes

    };

.eqea.util.asof.ajTickDataToInput

Efficient as-of join, joining an input data set to tick-data.

It is intended to be used for complex analytics, which are incompatible with .eqea.util.asof.ajFromCfg.

Parameters:

Name Type Description
inputData table Table containing all timestamps and instrumentIDs of interest. This inputData should have contain columns with names & types matching the primary time and sym columns of the target tick-data.
tickTabName symbol The name of the tick data table we wish to use in our as of join.
tickDataCols symbol[] The column(s) from tabName that we wish to include in our output. Example: If you wanted to get the price and size of a Trade at each time of your input data then tickDataCols would be pricesize
wc parse tree Where clause to apply to tick data.

Returns:

Type Description
table Result of our as-of join.

.eqea.util.pivotByCfg

this function pivots a dataset, using the contents of .eqea.analytics.cfg to define output column names.

Ii is intended to be used to pivot data produced by .eqea.util.asof.ajTickDataToInput.

Parameters:

Name Type Description
data table Table containing Order data which has been asof joined to tick data.
cfg table Subset of .eqea.analytics.cfg - contains all details required for current batch of analytics.
timeCol symbol Time column from the Order table. This is used in conjuction with idCol to perform the pivot.
idCol symbol Sym column from the Order table. This is used in conjuction with timeCol to perform the pivot.
pivValues symbol All columns from tick data containing the data that needs to be pivotted.

Returns:

Type Description
table Pivotted table that can be joined to our OrderAnalytics table.

.eqea.util.timeRackFromCfg

This function creates a time rack table from inputData based on the joinTimeOffset column from .eqea.analytics.cfg.

The time rack table is generated using joinOrderTimeCol + exec distinct joinTimeOffset from cfg where marketDataTabName=tickTabName.

It is intended to create a table of timestamps and instrumentIDs as required as an input in .eqea.util.asof.ajTickDataToInput.

Parameters:

Name Type Description
inputData table Table we want to use for as of join. Usually Order data.
tickTabName symbol Name of the tick data we are joining to.
cfg table Subset of .eqea.analytics.cfg - contains all details required for current batch of analytics.
joinOrderTimeCol symbol Time column from the Order table. This data what we use as the start point for the time rack table.

Returns:

Type Description
table Table with all timestamps and corresponding instrumentIDs as defined by the joinTimeOffset in .eqea.analytics.cfg