Deploying the ICE Consolidated Feed Handler with kdb Insights SDK
The following section details how to install and run the KX ICE Consolidated Feed Handler using the kdb Insights SDK.
Architecture
Pre-Requisites
Before running the KX ICE Consolidated Feed Handler with the kdb Insights SDK, the following prerequisites must be met:
- docker is installed.
- docker-compose is installed.
- You have login credentials for the ICE Consolidated Feed.
- You have login credentials for docker registry at portal.dl.kx.com.
- You have a kdb Insights license file (
kc.lic
).
Login to Docker Registry
To access the images needed for this walkthrough, login to the KX Docker registry:
docker login portal.dl.kx.com -u <insert registry username> -p <insert registry password>
Download the ICE kdb Insights SDK Example
To download the ICE kdb Insights SDK example, run:
version=1.0.1;\
curl -L https://portal.dl.kx.com/assets/raw/kxi-ice-fh/${version}/ice-insights-sdk-example-${version}.tar.gz -o ice-insights-sdk-example.tgz
Extract (untar) the files and navigate to the resulting directory:
tar -xvzf ice-insights-sdk-example.tgz;\
cd ice-insights-sdk-example
Configure the ICE kdb Insights SDK Example
Within the ice-insights-sdk-example directory, locate the docker-compose-one-node.yaml
file and the env
file. Update the following in the env
file:
- Update the kx_licence_dir variable to the directory path where your kdb Insights license file is located.
- Update the KXFEED_ICE_USERNAME and KXFEED_ICE_PASSWORD variables with your ICE username and password.
File Permissions
To avoid permission issues with containers mounting directories, run the following within the ice-insights-sdk-example
directory. This command may require sudo
access:
chmod -R 777 */
Or, use sudo
if necessary:
sudo chmod -R 777 */
Deploy the ICE kdb Insights SDK Example
After updating the configuration, deploy the ICE kdb Insights SDK example:
docker-compose --env-file env -f docker-compose-one-node.yaml up
Querying the Data
After deploying, open a second terminal to query the Quote
data published by the ICE feed using this getData REST call:
START=$(date "+%Y.%m.%dD00:00:00.000000000");\
END=$(date "+%Y.%m.%dD23:59:59.999999999");\
curl -X POST "localhost:8080/kxi/getData" -H "Content-Type: application/json" -H "Accepted: application/json" --data "{\"table\":\"Quote\",\"startTS\":\"$START\",\"endTS\":\"$END\"}"
To query the Trade
data, use:
START=$(date "+%Y.%m.%dD00:00:00.000000000");\
END=$(date "+%Y.%m.%dD23:59:59.999999999");\
curl -X POST "localhost:8080/kxi/getData" -H "Content-Type: application/json" -H "Accepted: application/json" --data "{\"table\":\"Trade\",\"startTS\":\"$START\",\"endTS\":\"$END\"}"
If neither call returns any data, review the logs to check for any issues.
Viewing the Logs
To check if the services have started successfully, view the logs using the following command within the ice-insights-sdk-example
directory:
docker-compose --env-file env -f docker-compose-one-node.yaml logs | less
If the ICE feed is publishing successfully, logs from the sp worker are displayed with messages such as, Received quote
or Received trade
.
If these messages are absent, search for ERROR
and FATAL
messages to identify any issues. If no errors are found, validate that it is market hours for the symbols listed in the KXFEED_ICE_SUBSCRIPTION_LIST
setting in the env
file.
Note
Logs also appear when you run docker-compose up
, but they can move quickly and be difficult to read. Use docker-compose logs
for easier reading and searching.
Tearing Down the ICE kdb Insights SDK Example
To teardown and delete the containers for the ICE kdb Insights SDK example, run the following command within the ice-insights-sdk-example
directory:
docker-compose --env-file env -f docker-compose-one-node.yaml down
To remove all data written to disk by the example, first delete all volumes:
docker volume rm docker_compose_rt-north-0;\
docker volume rm docker_compose_rt-north-session;\
docker volume rm docker_compose_rt-south-0;\
docker volume rm docker_compose_rt-south-session;
Then, remove all data written to mounted paths (requires sudo
access):
sudo bash -c 'rm -rf db/data/db/*; rm -rf db/data/logs/*; rm -rf log/*; rm -rf sp/checkpoints/*'