Skip to content

Order Ingest Pipeline

A pipeline has been provided with the KX Equities Analytics Accelerator to ingest order data, eqeaorderingest. This pipeline ingests .CSV files of historic data to the Order schema. Files for ingestion should be in .CSV format and the data should conform to the target Order schema. There is a configurable mapping of file header names to the Order schema column names. The column names and types of the Order schema are outlined below.

Customizing the Column Mapping

The column headers in the ingested csv file will be mapped to the Order schema column names as dictated by .eqea.orderColMap, which is defined in the equitiesEa.config.q file. The default mapping is

Csv file header Order schema column
strikeTime strikeTime
instrumentID instrumentID
OMS_ORDER_ID orderID
TRADE_DATE tradeDate
OMS_EXCHANGE_CODE exchangeID
MIC_CODE MIC
ISIN ISIN
TRADING_VENUE_CODE tradingVenueCode
EXECUTING_BROKER_CODE executingBrokerCode
PLACEMENT_QTY placementQty
EXECUTED_QTY executedQty
EXECUTED_AMT executedAmt
AVERAGE_PRICE averagePrice
LAST_EXECUTION_TIME lastExecutionTime
CANCEL_TIME cancelTime
ORDER_COMPLETED_TIME orderCompletedTime
FIX_SEND_TIME fixSendTime
PLACEMENT_DATETIME placementTime
LIMIT_PRICE limitPrice
ORDER_SIDE_CODE orderSideCode

If the column header names in the ingested .CSV differ from the above, then the names can be changed in the mapping. You can also remove rows from the mapping if the fields do not exist in the ingested .CSV. The mapping can be adjusted by editing the equitiesEa.config.q file. Refer to the following editing files section.

Customizing the Pipeline

The eqeaorderingest pipeline can be adapted to meet different requirements. By editing the eqeaorderingest-pipeline-spec.q file various aspects of the pipeline operation can be adjusted:

  • The pipeline reads a file from S3 by default. This can be changed to a different reader by editing the .fsi.reader variable. Refer to readers for the available readers.
  • The file targeted by the pipeline for ingest is configured by the .fsi.filePath variable. This should be updated to point toward the appropriate file for ingestion.
  • The region the file is located in is configured by the .fsi.region variable.

Refer to the following editing files section to edit the file.

Editing Files in the KX Equities Analytics Accelerator

In order to edit the equitiesEa.config.q or eqeaorderingest-pipeline-spec.q files, the KX Equities Analytics Accelerator must be unpacked. You can then edit them with a text editor. Then, the package should be pushed and deployed.

Unpack

Refer to the following documentation for general commands to unpack and re-package a package.

Edit

Now you can make the changes required to the equitiesEa.config.q or eqeaorderingest-pipeline-spec.q files.

Once changes have been made and packaged, the updated package must be pushed before being deployed.

Push and deploy

Refer to the following documentation on pushing a package.

Read the instructions to deploy an assembly.

Order Schema, Column Names and Types

name type
strikeTime timestamp
instrumentID symbol
orderID symbol
tradeDate date
exchangeID symbol
MIC symbol
ISIN symbol
tradingVenueCode symbol
executingBrokerCode symbol
placementQty long
executedQty long
executedAmt long
averagePrice float
lastExecutionTime timestamp
cancelTime timestamp
orderCompletedTime timestamp
limitPrice float
orderSideCode symbol

Order Ingest Normalisation

There is normalisation logic available in the order ingest pipeline to calculate the order complete time and strike time. This is disabled by default, but can be enabled by editing the eqeaorderingest-pipeline-spec.q file. Refer to the above steps to edit the file. To utilise the normalisation logic, the eqeaorderingest-pipeline-spec.q file should be edited to change the value of the .eqea.useTimezoneNormalization variable from false to true, resulting in that part of the file looking like:

// Variable to decide if we want to normalize orders or not
.eqea.useTimezoneNormalization:1b;

// Order normalisation configuration settings
if[.eqea.useTimezoneNormalization;
  // ingested Order time zone
  .eqea.orderTz:`$"America/New_York";
  .eqea.orderColMap:`strikeTime`STRIKE_TIME _.eqea.orderColMap;
  ];

Order time zone

The normalisation logic will convert all times in the ingested orders to UTC before calculating order complete time and strike time and then persisting the orders. The .eqea.orderTz variable should be edited to reflect the time zone of the ingested orders.