FIX Feed Handler on kdb Insights Enterprise

This page details how to install and run the KX FIX Feed Handler on kdb Insights Enterprise.

Architecture

KX FIX FH Architecture - Enterprise

Prerequisites

Before running the KX FIX Feed Handler on kdb Insights Enterprise, ensure the following prerequisites are met:

kdb Insights Enterprise is installed.
kxi CLI is installed.
helm CLI is installed.
Command line access to the Kubernetes cluster where your kdb Insights Enterprise install is running.
You have the login credentials for portal.dl.kx.com.

Setting up Kubernetes Secrets

Create a secret containing credentials for the KX Docker registry portal.dl.kx.com, as follows:

kubectl create secret -n <insert kubernetes namespace> docker-registry regcred --docker-username=<insert registry username> --docker-password=<insert registry password> --docker-server=portal.dl.kx.com

Downloading the Feed Handler Helm Chart

The Helm chart for the KX FIX Feed Handler is available on KX's public chart repository. Add this repository to your local Helm configuration by running the following:

helm repo add kx-ext https://portal.dl.kx.com/assets/helm --username <insert portal username> --password <insert portal password>

Update the repository to load the latest available charts:

helm repo update kx-ext

Use helm search to list the available versions of the KX FIX Feed Handler:

helm search repo kx-ext/rt-fix-pub

Select the latest version of the chart from the list of versions, then download it:

helm fetch kx-ext/rt-fix-pub --version  <insert latest version> --untar --devel

Note

Replace <insert latest version> with the latest version of the rt-fix-pub chart, for example 1.0.1.

This downloads and untars the rt-fix-pub chart in your current working directory.

Configuring the Feed Handler

The default configuration for the KX FIX Feed Handler is in the values.yaml file. To modify this configuration, create a copy, as follows:

cp rt-fix-pub/values.yaml my-values.yaml

Edit my-values.yaml to configure the Feed Handler.

The key sections in this configuration file are:

fix

The fix section is where all things related to receiving and processing FIX messages are configured. For detailed information on each setting, refer to the FIX Configuration documentation. The most important settings are described below:

sessionFile: The settings for each FIX session are configured in this file. The file can be found and edited in the config directory of the rt-fix-pub helm chart. Read the documentation here for full details on every setting. For a basic deployment, review the following settings in the file:
ConnectionType: Defaults to initiator. Update to acceptor if your FIX counterparty is going to be the initiator of the connection. For this example, it is left as initiator.
SenderCompID: Your ID as associated with this FIX session. For this example, it is set to KX.
TargetCompID: Counter parties ID as associated with this FIX session. For this example, it is set to SERVER.
BeginString: Version of FIX this session should use. Set to FIX.4.4 in this example.
DataDictionary: XML definition file for validating incoming FIX messages and processing repeating groups. Default dictionaries for each FIX version can be found in the config directory of the rt-fix-pub helm chart. In this example, FIX44.xml is used.
SocketConnectHost: Host to connect to when ConnectionType=initiator. For this example, we are connecting to a FIX data simulator with the host name fix-server.
SocketConnectPort: Port to connect to when ConnectionType=initiator. Set to 1235 in this example. Use SocketAcceptPort instead if ConnectionType=acceptor.
dbSchemaFile: Each table that is published downstream of the KX FIX Feed Handler is defined in this xml file. It can be found and edited in the config directory of the rt-fix-pub helm chart. For this example, a single Trade table is defined.
eventToTableFile: This file maps each table to a list of message types (FIX tag 35). In this example, the Trade table is mapped to the message type AE, which is published by the FIX data simulator being connected to.

ports

If ConnectionType=acceptor in your FIX session file, ensure the SocketAcceptPort is listed here. For example:

ports:
  - 1235

This allows the port to be accessible by other processes.

service

If ConnectionType=acceptor in your FIX session file, and the counterparty publishing to the KX FIX Feed Handler is not a pod deployed in the same kubernetes namespace, then you must use a service type other than ClusterIP in your values.yaml.

For all the major cloud providers (AWS, Azure, GCP), setting the type to LoadBalancer automatically provisions an external IP that your counterparty can use as the host name to connect to. Get this external IP by running kubectl get svc rt-fix-pub.

Note

The kubernetes service is created only if create: true and at least 1 port is listed under the ports section.

configMap

Any configuration files being used in the config directory of the helm chart, must be listed under the files section in your values.yaml. The files listed here must also exist in the config directory of the helm chart.

Each of the files listed are mounted by a config map that allows the KX FIX Feed Handler to read them from the directory /opt/kx/kxfeed_fix/config/.

By default the data dictionary FIX44.xml is listed here, along with the Session.cfg, eventToTable.json and schema.xml files. If you are using a different data dictionary ensure that:

The dictionary exists in the config directory of the helm chart.
The name of the dictionary file is listed under configMap.files in the values.yaml.
The path to the dictionary is set to /opt/kx/kxfeed_fix/config/NAME_OF_DICTIONARY_FILE.xml.

rt

In a kdb Insights Enterprise deployment, typically the KX FIX Feed Handler publishes data to an RT stream. This section configures the interaction between an RT stream and the KX FIX Feed Handler. For detailed information on each setting, refer to the RT documentation. For this walkthrough, ensure the following setting is correct:

sinkName: Set this to the name of the RT stream for publishing. For example, set it to rt-fix-example-package-fix-north for the example package discussed further down.

imagePullSecrets

The imagePullSecrets section is for listing the names of secrets required by the KX FIX Feed Handler. The name of the secret created in Setting up Kubernetes Secrets should be listed here:

imagePullSecrets:
  - name: regcred

Deploying a kdb Insights Enterprise App

To get started with ingesting and querying FIX data, download the example kdb Insights Enterprise package by running the following:

version=1.0.1;\
curl -L -O https://portal.dl.kx.com/assets/raw/kxi-fix-fh/${version}/fix-example-package-${version}.kxi

Use this package as the base to build your own new package, or take elements from it and integrate them into an existing package.

To deploy the package run the following set of commands:

user@demo-host:~/fix-example$ ls
fix-example-package-1.0.1.kxi
user@demo-host:~/fix-example$
user@demo-host:~/fix-example$ kxi package push fix-example-package/1.0.1
Pushing: fix-example-package-1.0.1.kxi
{
    "fix-example-package": [
        {
            "version": "1.0.1",
            "_status": "InstallationStatus.SUCCESS"
        }
    ]
}
user@demo-host:~/fix-example$
user@demo-host:~/fix-example$ kxi package deploy fix-example-package/1.0.1
user@demo-host:~/fix-example$

Deploying the Feed Handler

Deploy the KX FIX Feed Handler using the my-values.yaml file created and edited earlier.

Reminder

For the feed handler to publish successfully to the fix-example-package, myvalues.yaml must have sinkName: rt-fix-example-package-fix-north under the stream: section.

user@demo-host:~/helm_charts/fix$ ls
my-values.yaml  rt-fix-pub
user@demo-host:~/helm_charts/fix$
user@demo-host:~/helm_charts/fix$ helm install rt-fix-pub rt-fix-pub -f my-values.yaml -n <insert namespace>
NAME: rt-fix-pub
LAST DEPLOYED: Tue Jun 11 14:58:55 2024
NAMESPACE: user
STATUS: deployed
REVISION: 1
TEST SUITE: None
user@demo-host:~/helm_charts/fix$

Querying the Data

Once the fix-example-package and KX FIX Feed Handler have been deployed successfully, navigate to the kdb Insights Enterprise UI and query the data.

kdb Insights Enterprise FIX Data Query

The data shown here was published to the KX FIX Feed Handler by an internal FIX data simulator. Here are the raw FIX messages that it reads and publishes on loop:

8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:01.000|571=TR12346|570=Y|55=GOOGL|32=950|31=151.00|75=20241025|60=20241025-15:29:01.000|552=2|54=1|37=123457|54=2|37=654322|10=124|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:02.000|571=TR12347|570=N|55=MSFT|32=500|31=149.75|75=20241025|60=20241025-15:29:02.000|552=2|54=1|37=123458|54=2|37=654323|10=125|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:03.000|571=TR12348|570=Y|55=AAPL|32=1200|31=150.25|75=20241025|60=20241025-15:29:03.000|552=2|54=1|37=123459|54=2|37=654324|10=126|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:04.000|571=TR12349|570=N|55=TSLA|32=400|31=752.80|75=20241025|60=20241025-15:29:04.000|552=2|54=1|37=123460|54=2|37=654325|10=127|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:05.000|571=TR12350|570=Y|55=AMZN|32=300|31=3290.75|75=20241025|60=20241025-15:29:05.000|552=2|54=1|37=123461|54=2|37=654326|10=128|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:06.000|571=TR12351|570=Y|55=AAPL|32=1050|31=150.65|75=20241025|60=20241025-15:29:06.000|552=2|54=1|37=123462|54=2|37=654327|10=129|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:07.000|571=TR12352|570=N|55=FB|32=700|31=340.50|75=20241025|60=20241025-15:29:07.000|552=2|54=1|37=123463|54=2|37=654328|10=130|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:08.000|571=TR12353|570=Y|55=GOOGL|32=450|31=152.10|75=20241025|60=20241025-15:29:08.000|552=2|54=1|37=123464|54=2|37=654329|10=131|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:09.000|571=TR12354|570=Y|55=MSFT|32=650|31=150.15|75=20241025|60=20241025-15:29:09.000|552=2|54=1|37=123465|54=2|37=654330|10=132|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:10.000|571=TR12355|570=N|55=AAPL|32=1000|31=150.55|75=20241025|60=20241025-15:29:10.000|552=2|54=1|37=123466|54=2|37=654331|10=133|

Viewing the Feed Handler Logs

View the KX FIX Feed Handler logs using kubectl:

kubectl logs rt-fix-pub-0 | less

To view the FIX message logs, shell into the pod and go to the /kxfeed_fix/log/ directory:

user@demo-host:~$ kubectl exec -it rt-fix-pub-0 -- /bin/bash
bash-5.1$
bash-5.1$ cd /kxfeed_fix/log/
bash-5.1$ ls
FIX.4.4-KX-SERVER.event.current.log  FIX.4.4-KX-SERVER.messages.current.log  GLOBAL.event.current.log  GLOBAL.messages.current.log
bash-5.1$

Tear down the Feed Handler

To tear down (uninstall) the KX FIX Feed Handler, run:

user@demo-host:~/helm_charts/fix$ helm uninstall rt-fix-pub -n <insert namespace>
release "rt-fix-pub" uninstalled
user@demo-host:~/helm_charts/fix$

Tear down the Example Package

To tear down the fix-example-package, run:

kxi package teardown fix-example-package/1.0.1

To delete all data associated with the package, run:

kxi package teardown --rm-data fix-example-package/1.0.1

Warning

The --rm-data flag erases all the data in the specified package's database. Use with caution.