Skip to content

Deploy the ACTIV Market Data Feed Handler with kdb Insights Enterprise

This page details how to install the KX ACTIV Market Data Feed Handler on kdb Insights Enterprise from its Helm chart and query the data it publishes.

The ACTIV Feed Handler streams real-time market data from the ACTIV One Platform into kdb Insights Enterprise, where you can store and query it.

ACTIV deployment process with kdb Insights Enterprise overview

To configure and deploy the feed handler with kdb Insights Enterprise and query the data it publishes, complete the following tasks:

Architecture

The feed handler runs as a Kubernetes StatefulSet in the same Kubernetes namespace as kdb Insights Enterprise. It publishes ACTIV updates to the ingest stream north in kdb Insights Reliable Transport (RT), the messaging layer that carries data between components. The stream processor reads from north and writes the processed data to the stream south, and the storage manager and data access services store the data from south.

The following diagram shows how the feed handler publishes to kdb Insights Enterprise:

Feed handler publishes ACTIV data over RT into kdb Insights Enterprise, which stores it and serves queries.

Prerequisites

Before running the ACTIV Feed Handler on kdb Insights Enterprise, make sure you have the following:

Set up Kubernetes secrets

Create a secret containing your credentials for the KX Docker registry portal.dl.kx.com:

kubectl create secret -n <insert kubernetes namespace> docker-registry regcred --docker-username=<insert registry username> --docker-password=<insert registry password> --docker-server=portal.dl.kx.com

Download the feed handler Helm chart

The Helm chart for the ACTIV Feed Handler is available on KX's public chart repository. Add this repository to your local Helm configuration by running the following:

helm repo add kx-ext https://portal.dl.kx.com/assets/helm --username <insert portal username> --password <insert portal password>

Update the repository to load the latest available charts:

helm repo update kx-ext

Use helm search to list the available versions of the ACTIV Feed Handler:

helm search repo kx-ext/rt-activ-pub

Select the latest version of the chart from the list of versions, and then download it:

helm fetch kx-ext/rt-activ-pub --version <insert latest version> --untar --devel

Note

Replace <insert latest version> with the latest version of the rt-activ-pub chart, for example 1.15.3.

This downloads and untars the rt-activ-pub chart in your current working directory.

Configure the feed handler

The default configuration for the ACTIV Feed Handler is in the values.yaml file. To modify this configuration, create a copy, as follows:

cp rt-activ-pub/values.yaml my-values.yaml

Edit my-values.yaml to configure the feed handler.

Key sections in this configuration file include:

activ

Contains the settings for configuring the ACTIV Feed Handler. For detailed information on each setting, refer to the ACTIV configuration documentation. Ensure the following settings are correct:

Setting Description
host The ACTIV server hostname.
userId Your ACTIV user ID.
pass Your ACTIV password.
subscription What to subscribe to. With enableQueries: "false", list plain symbols separated by spaces, for example "SHEL.L POW.TO". With enableQueries: "true", list ACTIV queries separated by ; with no spaces, for example "symbol=SHEL.L;symbol=POW.TO". Queries can use wildcards, for example "symbol=GOOGL*". See Subscriptions.
enableQueries Set to "false" to subscribe to plain symbols. The chart default is "true", which reads each ;-separated entry as an ACTIV query. The chart default for subscription is a space-separated list of plain symbols, so change one of the two settings.
subscriptionFile Set to /tmp/kxfeed_activ/symbols.txt. The startup script writes the subscription entries to symbols.txt in tmpConfigDir (/tmp/kxfeed_activ by default), but the chart default for subscriptionFile points to a different file, so subscription has no effect unless you change this setting.
pubDelay The publish delay in milliseconds. The default is 100. Set to 0 to turn batching off.

For example, to subscribe to SHEL.L and every symbol that starts with GOOGL:

activ:
  subscription:     "symbol=SHEL.L;symbol=GOOGL*"
  enableQueries:    "true"
  subscriptionFile: "/tmp/kxfeed_activ/symbols.txt"

Warning

pass is stored in my-values.yaml in plain text. Keep this file out of source control and restrict who can read it.

stream

In a kdb Insights Enterprise deployment, the ACTIV Feed Handler publishes data to an RT stream.

Setting Description
sinkName Set this to the name of the RT stream for publishing. For example, set it to rt-activ-example-package-activ-north for the example package in Deploy a kdb Insights Enterprise app below.

The chart default is rt-activ-assembly-north, which doesn't match the example package, so you must change it.

rt

The rt section configures the RT publisher inside the feed handler. For detailed information on each setting, refer to the RT settings. The defaults are suitable for most deployments.

persistence

The RT publisher stores its local log files on a Persistent Volume Claim (PVC). Set storageSize to a value that suits the volume of data you expect to ingest. See persistence.

imagePullSecrets

The imagePullSecrets section lists the names of the secrets required to pull the feed handler image. Add the secret created in Set up Kubernetes secrets:

imagePullSecrets:
  - name: regcred

Deploy a kdb Insights Enterprise app

To get started with ingesting and querying ACTIV market data, download the example kdb Insights Enterprise package by running the following:

export BEARER=<your bearer token>;\
version=1.15.3;\
curl -L -O --oauth2-bearer "$BEARER" https://portal.dl.kx.com/assets/raw/kxi-activ-fh/${version}/activ-example-package-${version}.kxi

Portal login

Downloads from the KX portal need a login. Sign in at portal.dl.kx.com, select your email address in the top right to open Token Management, and select Add New Bearer Token. Set BEARER to that token before you run the command. Use the token for downloads from your own machine only, not in a deployment.

Use this package as the base to build your own new package, or take elements from it and integrate them into an existing package.

To deploy the package, run the following commands:

kxi pm push activ-example-package-1.15.3.kxi
kxi pm deploy activ-example-package/1.15.3

In this example, the package expects the default ACTIV Feed Handler configuration. Change only the configuration fields discussed in Configure the feed handler.

Deploy the feed handler

Deploy the ACTIV Feed Handler using the my-values.yaml file created and edited in the Configure the feed handler section.

For the feed handler to publish successfully to the activ-example-package, my-values.yaml must have sinkName: rt-activ-example-package-activ-north under the stream: section.

helm install rt-activ-pub rt-activ-pub -f my-values.yaml -n <insert namespace>

Check that the feed handler is running:

kubectl get pods -n <insert namespace> | grep activ

The pod rt-activ-pub-0 shows 1/1 Running.

Query the data

After you deploy the activ-example-package and the ACTIV Feed Handler, kdb Insights Enterprise stores the data in the package's activdb database, in the following tables:

Table Contains
quote Best bid and offer quotes.
trade Trades.
cancor Trade cancellations and corrections. This table fills only when ACTIV sends a cancellation or correction, so it is often empty.

The columns match the published tables. The pipeline in the package also adds an updateTS column with the time each row was ingested.

Query the tables with any of the kdb Insights Enterprise query methods, for example the Preview API in the web interface with quote, trade, or cancor as the table name.

If no data appears, check that the market is open for your symbols. A subscription returns an initial snapshot at any time, but trades and quotes arrive only while the market is trading.

View the feed handler logs

View the ACTIV Feed Handler logs using kubectl:

kubectl logs rt-activ-pub-0 -n <insert namespace> | less

The feed handler also writes its own log to the path set by activ.logFile in values.yaml. The default is /tmp/kxfeed_activ.log inside the container, which isn't on the PVC, so the file is lost when the pod restarts. The RT publisher logs are separate. They are stored on the PVC under rt.logPath.

Tear down the feed handler

To tear down (uninstall) the ACTIV Feed Handler, run the following command:

helm uninstall rt-activ-pub -n <insert namespace>

Kubernetes sends the process a SIGTERM and waits for a grace period (about 30 seconds) before it stops the pod.

Tear down the example package

To tear down the activ-example-package, run:

kxi pm teardown activ-example-package/1.15.3

To delete all data associated with the package, run:

kxi pm teardown --rm-data activ-example-package/1.15.3

Warning

The --rm-data flag erases all data in the specified package's database. Use with caution.

Next steps