Deploying the FIX Feed Handler with kdb Insights SDK
This page details how to install and run the KX FIX Feed Handler using the kdb Insights SDK.
Architecture
Prerequisites
Before running the KX FIX Feed Handler with the kdb Insights SDK, the following prerequisites must be met:
- docker is installed.
- docker-compose is installed.
- You have login credentials for docker registry at portal.dl.kx.com.
- You have a kdb Insights license file (
kc.lic
).
Login to Docker registry
To access the images needed for this walkthrough, login to the KX Docker registry:
docker login portal.dl.kx.com -u <insert registry username> -p <insert registry password>
Download the FIX kdb Insights SDK example
To download the FIX kdb Insights SDK example, run:
version=1.0.1;\
curl -L https://portal.dl.kx.com/assets/raw/kxi-fix-fh/${version}/fix-insights-sdk-example-${version}.tar.gz -o fix-insights-sdk-example.tgz
Extract (untar) the files and navigate to the resulting directory:
tar -xvzf fix-insights-sdk-example.tgz;\
cd fix-insights-sdk-example
Configure the FIX kdb Insights SDK example
Within the fix-insights-sdk-example directory, locate the docker-compose-one-node.yaml
file and the env
file. Update the following in the env
file:
- Update the
kx_licence_dir
variable to the directory path where your kdb Insights license file is located.
Next, locate the Session.cfg
file that can be found in the cfg
directory. Update the following in this file:
- Update the connection settings to that of the counterparty that publishes to KX FIX Feed Handler. In our example, the counterparty being connected to is an internal FIX data simulator.
The FIX kdb Insights SDK example is setup to ingest 1 message type AE
and has a singular Trade
table defined for storing these messages in the kdb Insights database. If you wish to changes this, edit the following settings in the fix-insights-sdk-example directory:
- cfg/schema.xml: The KX FIX Feed Handler uses this config to parse messages. You can add new tables or edit the existing
Trade
table. See Schema Config for more details. - cfg/eventToTable.json: The KX FIX Feed Handler uses this config to map tables defined in the
schema.xml
to message types. Read more about theeventToTableFile
setting here. - db/config/assembly.yaml: The
table
section in this config contains the definitions of the tables in the kdb Insights database. Similar to theschema.xml
file, you can add new tables or edit the existingTrade
table. See more details on kdb Insights database schema configuration here. - sp/spec.q: Editing the
Trade
table or adding new tables can require updates to the kdb Insights stream processor pipeline, which is responsible for writing the data coming from the KX FIX Feed Handler to the kdb Insight database. See here for more details on writing and running pipelines.
File permissions
To avoid permission issues with containers mounting directories, run the following within the fix-insights-sdk-example
directory. This command may require sudo
access:
chmod -R 777 */
Or, use sudo
if necessary:
sudo chmod -R 777 */
Deploy the FIX kdb Insights SDK example
After updating the configuration, deploy the FIX kdb Insights SDK example:
docker-compose --env-file env -f docker-compose-one-node.yaml up
Query the data
After deploying, open a second terminal to query the data. In our example the internal FIX data simulator that we had the KX FIX Feed Handler connect to published the following FIX messages on loop:
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:01.000|571=TR12346|570=Y|55=GOOGL|32=950|31=151.00|75=20241025|60=20241025-15:29:01.000|552=2|54=1|37=123457|54=2|37=654322|10=124|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:02.000|571=TR12347|570=N|55=MSFT|32=500|31=149.75|75=20241025|60=20241025-15:29:02.000|552=2|54=1|37=123458|54=2|37=654323|10=125|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:03.000|571=TR12348|570=Y|55=AAPL|32=1200|31=150.25|75=20241025|60=20241025-15:29:03.000|552=2|54=1|37=123459|54=2|37=654324|10=126|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:04.000|571=TR12349|570=N|55=TSLA|32=400|31=752.80|75=20241025|60=20241025-15:29:04.000|552=2|54=1|37=123460|54=2|37=654325|10=127|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:05.000|571=TR12350|570=Y|55=AMZN|32=300|31=3290.75|75=20241025|60=20241025-15:29:05.000|552=2|54=1|37=123461|54=2|37=654326|10=128|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:06.000|571=TR12351|570=Y|55=AAPL|32=1050|31=150.65|75=20241025|60=20241025-15:29:06.000|552=2|54=1|37=123462|54=2|37=654327|10=129|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:07.000|571=TR12352|570=N|55=FB|32=700|31=340.50|75=20241025|60=20241025-15:29:07.000|552=2|54=1|37=123463|54=2|37=654328|10=130|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:08.000|571=TR12353|570=Y|55=GOOGL|32=450|31=152.10|75=20241025|60=20241025-15:29:08.000|552=2|54=1|37=123464|54=2|37=654329|10=131|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:09.000|571=TR12354|570=Y|55=MSFT|32=650|31=150.15|75=20241025|60=20241025-15:29:09.000|552=2|54=1|37=123465|54=2|37=654330|10=132|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:10.000|571=TR12355|570=N|55=AAPL|32=1000|31=150.55|75=20241025|60=20241025-15:29:10.000|552=2|54=1|37=123466|54=2|37=654331|10=133|
This data was published to the Trade
table by the KX FIX Feed Handler. We can query this data using this getData REST call:
START=$(date "+%Y.%m.%dD00:00:00.000000000");\
END=$(date "+%Y.%m.%dD23:59:59.999999999");\
curl -X POST "localhost:8080/kxi/getData" -H "Content-Type: application/json" -H "Accepted: application/json" --data "{\"table\":\"Trade\",\"startTS\":\"$START\",\"endTS\":\"$END\"}"
If the call does not return any data, review the logs to check for any issues.
View the logs
To check if the services have started successfully, view the logs using the following command within the fix-insights-sdk-example
directory:
docker-compose --env-file env -f docker-compose-one-node.yaml logs | less
If the FIX data is being published successfully, logs from the sp worker are displayed with messages such as, Received trade
.
Note
Logs also appear when you run docker-compose up
, but they can move quickly and be difficult to read. Use docker-compose logs
for easier reading and searching.
Tear Down the FIX kdb Insights SDK example
To tear down and delete the containers for the FIX kdb Insights SDK example, run the following command within the fix-insights-sdk-example
directory:
docker-compose --env-file env -f docker-compose-one-node.yaml down
To remove all data written to disk by the example, first delete all volumes:
docker volume rm fix-insights-sdk-example_rt-north-0;\
docker volume rm fix-insights-sdk-example_rt-north-session;\
docker volume rm fix-insights-sdk-example_rt-south-0;\
docker volume rm fix-insights-sdk-example_rt-south-session;
Then, remove all data written to mounted paths (requires sudo
access):
sudo bash -c 'rm -rf db/data/db/*; rm -rf db/data/logs/*; rm -rf log/*; rm -rf store/*; rm -rf sp/checkpoints/*'