Deploying the ICE Consolidated Feed Handler with kdb Insights Enterprise
The following section details how to install and run the KX ICE Consolidated Feed Handler on kdb Insights Enterprise.
Architecture
Prerequisites
Before running the KX ICE Consolidated Feed Handler on kdb Insights Enterprise, ensure the following prerequisites are met:
- kdb Insights Enterprise is installed.
kxi
CLI is installed.helm
CLI is installed.- Command line access to the Kubernetes cluster where your kdb Insights Enterprise install is running.
- Login credentials for the ICE Consolidated Feed.
- Login credentials for portal.dl.kx.com.
Setting up Kubernetes Secrets
First, create a Kubernetes secret containing your ICE login credentials:
kubectl create secret generic my-ice-secrets --from-literal=my-username=<insert ice username> --from-literal=my-password=<insert ice password> -n <insert kubernetes namespace>
Then, create a secret containing your credentials for the KX Docker registry portal.dl.kx.com :
kubectl create secret -n <insert kubernetes namespace> docker-registry regcred --docker-username=<insert registry username> --docker-password=<insert registry password> --docker-server=portal.dl.kx.com
Downloading the Feed Handler Helm Chart
The Helm chart for the KX ICE Consolidated Feed Handler is available on KX's public chart repository. Add this repository to your local Helm configuration by running the following:
helm repo add kx-ext https://portal.dl.kx.com/assets/helm --username <insert portal username> --password <insert portal password>
Update the repository to load the latest available charts:
helm repo update kx-ext
Use helm search
to list the available versions of the KX ICE Consolidated Feed Handler:
helm search repo kx-ext/rt-ice-pub
Select the latest version of the chart from the list of versions, then download it:
helm fetch kx-ext/rt-ice-pub --version <insert latest version> --untar --devel
Note
Replace <insert latest version>
with the latest version of the rt-ice-pub
chart, for example 1.0.1
.
This downloads and untars the rt-ice-pub
chart in your current working directory.
Configuring the Feed Handler
The default configuration for the KX ICE Consolidated Feed Handler is in the values.yaml
file. To modify this configuration, create a copy:
cp rt-ice-pub/values.yaml my-values.yaml
Edit my-values.yaml
to configure the Feed Handler.
Key sections in this configuration file include:
ice
Contains the settings for configuring the KX ICE Consolidated Feed Handler. For detailed information of each setting, refer to the ICE Configuration documentation. Ensure the following settings are correct:
- primaryConnection: Verify the ICE primary connection host/ports.
- backupConnection: Verify the ICE backup connection host/ports.
- secrets: Ensure the name of the ICE credentials secret matches the one created in Setting up Kubernetes Secrets. Leave the
username
andpassword
settings as is.
rt
In a kdb Insights Enterprise deployment, typically the KX ICE Consolidated Feed Handler publishes data to an RT stream. This section configures the interaction between an RT stream and the KX ICE Consolidated Feed Handler. For detailed information of each setting, refer to the RT documentation. Ensure the following setting is correct:
- sinkName: Set this to the name of the RT stream for publishing. For example, set it to
rt-ice-example-package-ice-north
for the example package discussed later.
imagePullSecrets
The imagePullSecrets
section is for listing the names of secrets required by the KX ICE Consolidated Feed Handler. The names of both secrets created in Setting up Kubernetes Secrets should be displayed here:
imagePullSecrets:
- name: regcreds
- name: my-ice-secrets
Deploying a kdb Insights Enterprise App
To get started with ingesting and querying ICE market data, download the example kdb Insights Enterprise package by running the following:
version=1.0.1;\
curl -L -O https://portal.dl.kx.com/assets/raw/kxi-ice-fh/${version}/ice-example-package-${version}.kxi
Use this package as the base to build your own new package, or take elements from it and integrate them into an existing package.
To deploy the package run the following set of commands:
user@demo-host:~/ice-example$ ls
ice-example-package-1.0.1.kxi
user@demo-host:~/ice-example$
user@demo-host:~/ice-example$ kxi package push ice-example-package/1.0.1
Pushing: ice-example-package-1.0.1.kxi
{
"ice-example-package": [
{
"version": "1.0.1",
"_status": "InstallationStatus.SUCCESS"
}
]
}
user@demo-host:~/ice-example$
user@demo-host:~/ice-example$ kxi package deploy ice-example-package/1.0.1
user@demo-host:~/ice-example$
Note
This example kxi
package expects the default KX ICE Consolidated Feed Handler configuration. Only the configuration fields discussed in Configuring the Feed Handler should be changed.
Deploying the Feed Handler
Deploy the KX ICE Consolidated Feed Handler using the my-values.yaml
file created and edited earlier.
!!! Reminder
For the feed handler to publish successfully to the ice-example-package
, myvalues.yaml
must have sinkName: rt-ice-example-package-ice-north
under the stream:
section.
user@demo-host:~/helm_charts/ice$ ls
my-values.yaml rt-ice-pub
user@demo-host:~/helm_charts/ice$
user@demo-host:~/helm_charts/ice$ helm install rt-ice-pub rt-ice-pub -f my-values.yaml -n <insert namespace>
NAME: rt-ice-pub
LAST DEPLOYED: Tue Jun 11 14:58:55 2024
NAMESPACE: user
STATUS: deployed
REVISION: 1
TEST SUITE: None
user@demo-host:~/helm_charts/ice$
Querying the Data
Once the ice-example-package
and KX ICE Consolidated Feed Handler have been deployed successfully, navigate to the kdb Insights Enterprise UI and query the data.
Viewing the Feed Handler Logs
View the KX ICE Consolidated Feed Handler logs using kubectl
:
kubectl logs rt-ice-pub-0 | less
Teardown the Feed Handler
To teardown (uninstall) the KX ICE Consolidated Feed Handler, run:
user@demo-host:~/helm_charts/ice$ helm uninstall rt-ice-pub -n <insert namespace>
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/lbritton/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/lbritton/.kube/config
release "rt-ice-pub" uninstalled
user@demo-host:~/helm_charts/ice$
Teardown the Example Package
To teardown the ice-example-package
, run:
kxi package teardown ice-example-package/1.0.1
To delete all data associated with the package, run:
kxi package teardown --rm-data ice-example-package/1.0.1
Warning
The --rm-data
flag erases all data in the specified package's database. Use with caution.