Skip to content

ICE Equities Analytics Accelerator Quickstart Guide

The ICE Equities Analytics is an FSI Accelerator package which comes with ready made Pipelines, Views, and Schemas out of the box.

Prerequisites

This quickstart guide is not intended to be a comprehensive KDB Insights Install Guide, for more information regarding KDB Insights installation refer to the kdb Insights Enterprise documentation). This guide assumes the following pre-requisites:

  • KDB Insights Enterprise installed
  • Credentials obtained and configured:
    • Insights Users:
      • GUI User
      • API Client
  • KX Downloads Portal bearer token to download packages/charts (represented by BEARER in this guide).
  • Tools Used:
    • Access to *nix command-line
    • KDB Insights CLI - kxi
  • Kubernetes tools:
    • kubectl
    • K9s
  • Helm installed and logged in to nexus
  • VSCode plugins:
    • KX kdb
    • Jupyter

Quick Start Guide

Download the required FSI packages from the KX Downloads Portal:

# Download fsi-lib - the Core FSI Library
curl -s --fail-with-body -D /dev/stderr --oauth2-bearer ${BEARER} -L -OJ https://portal.dl.kx.com/assets/raw/kxi-accelerators/fsi/fsi-lib/packages/1.1.3/fsi-lib-1.1.3.kxi

# Download fsi-app-ice-eqea - the ICE Equities Analysis Accelerator
curl -s --fail-with-body -D /dev/stderr --oauth2-bearer ${BEARER} -L -OJ https://portal.dl.kx.com/assets/raw/kxi-accelerators/fsi/fsi-ice-eqea/packages/1.0.0/fsi-app-ice-eqea-1.0.0.kxi

Unpack

Unpack fsi-app-ice-eqea:

kxi package unpack fsi-app-ice-eqea-1.0.0.kxi

Configure

Configure your pipelines to target files. In this quickstart, we configure the minimum set of pipelines. For full functionality, refer to Additional pipeline configuration.

Order ingest

First, we configure the Order ingest pipeline.

If we are using normalisation, the setting is turned on by setting the boolean as true. However, in this example it is set as false:

sed -i 's|\.eqea\.useTimezoneNormalization\:.*|.eqea.useTimezoneNormalization:0b;|g' fsi-app-ice-eqea/src/eqeaorderingest-pipeline-spec.q

By not activating normalisation, our orders are ingested as-is, without a strike time calculation or timezone adjustment.

Set the target file:

sed -i 's|\.fsi\.filePath\:.*|.fsi.filePath:`$":s3://my-bucket/order_data.csv";|g' fsi-app-ice-eqea/src/eqeaorderingest-pipeline-spec.q

fxRates

If you are using fxRates to calculate values in USD, set the target file as follows:

sed -i 's|\.fsi\.filePath\:.*|.fsi.filePath:`$":s3://my-bucket/fxRates.csv";|g' fsi-app-ice-eqea/src/eqeafxrates-pipeline-spec.q

fxRates are required to calculate the value USD, which is the foundation of the dashboard metrics.

Market data

Market data for the time period the order covers is required.

We set the filepath in the ICE market data ingestion pipeline to the corresponding ICE replay file:

sed -i 's|\.fsi\.filePath\:.*|.fsi.filePath:`$":s3://my-bucket/my-ice-replay.txt";|g' fsi-app-ice-eqea/src/iceEqHistoricReplay-pipeline-spec.q

This pipeline ingests both Trades and Quotes.

Pack and deploy

Pack and deploy the packages: 

kxi package packit fsi-app-ice-eqea --tag --version 1.0.0

Push the packages to Insights and Deploy the fsi-app-ice-eqea assembly:

# Install the packages
kxi package push --force fsi-lib/1.1.3
kxi package push --force fsi-app-ice-eqea/1.0.0
# Deploy the assembly
kxi package deploy fsi-app-ice-eqea/1.0.0

Note

At this point you should be able to see the pipelines booting up on the GUI or on k9s.

Access APIs

Once deployed, we can query the tables using getData or getTicks, then use the generation API to look at what the order analytics would look like.

Run the generation pipeline to persist the order analytics

Now that data is ingested and the API for generation is working, we can run an instance of the nightly generation pipeline that persists the order analytics. Refer to the guide for generation and persistence of orderAnalytics.

View summary of order analytics

Once orderAnalytics has been written in to the database, we can now query the table and use the summary rollup functions. These summary rollups power the example dashboards.

API

Call the summary API on the orderAnalytics table for rollup aggregations using the API.

Dashboards

Log into the dashboards to investigate order analytics. The dashboard is named Equity-Execution-Analysis.

The dashboard serves as an example of rendering some metrics from the summary rollup calculations.

Additional pipeline configuration

To use the full feature set of the accelerator, the following additional reference pipelines are also required:

Exchange reference data

If normalising Order data, exchange reference data is required to calculate strike time:

sed -i 's|\.fsi\.filePath\:.*|.fsi.filePath:`$":s3://my-bucket/exchangeHours.csv";|g' fsi-app-ice-eqea/src/eqeaexchange-refdata-pipeline-spec.q

Composite mapping

If composite tickers are being used for market data but orders are for local tickers, then we must ingest a composite ticker map:

sed -i 's|\.fsi\.filePath\:.*|.fsi.filePath:`$":s3://my-bucket/compositeTickerMap.csv";|g' fsi-app-ice-eqea/src/eqeacompositeticker-pipeline-spec.q

This feature also requires a boolean setting turned on (it is enabled by default) in a custom file in an overlay:

.eqea.mapToCompositeTicker:1b;

Next steps - nightly schedules

Once happy with all pipelines, the next step is to set up a re-occuring schedule of ingest pipelines to consume all data nightly, followed by a pipeline that generates the analytics.

Refer to the scheduling pipelines documentation.