Skip to content

Kubernetes Reference Deployment

This page provides reference helm charts for the kdb Insights Database and Reliable Transport (RT), for Insights SDK customers using Kubernetes. The deployment includes additional steps to publish data to the database before querying it back.

This guide is aimed at users who wish to develop APIs that can be run against their data. kdb Insights supports several interfaces, including:

  1. C
  2. Java
  3. q
  4. Python

Prerequisites

Ensure the following are installed before attempting the quickstart steps:

  1. helm
  2. kubectl

Provide a license

A license for kdb+ Cloud Edition is required. It can be generated from a valid kc.lic file with base64 encoding.

base64 -w0 /<path to kc.lic>/kc.lic > license.txt
kubectl create secret generic kx-license --from-file=license=./license.txt

If you haven't already done so, sign up for a trial kdb Insights SDK license on the KX website.

KX Downloads Portal

You need to have credentials to the KX Downloads Portal before you can pull down the images and obtain a bearer token.

The trial download process on the KX website, provides a bearer token that can be used.

If you have an account for the KX Downloads Portal, then the bearer token can be obtained as follows

  1. Log into the KX Download Portal.
  2. Once authenticated, navigate to https://portal.dl.kx.com/auth/token.
  3. Click Add New Bearer token.
  4. Click Copy Bearer.

The steps below create a kubernetes secret.

REGISTRY=portal.dl.kx.com
USER=joe@kx.com
PASS=<bearer token>
kubectl create secret docker-registry regcred-portal --docker-username=$USER --docker-password=$PASS --docker-server=$REGISTRY

Quickstart

Download

  1. Ensure you have access to the appropriate repo for the charts. 

    $ helm repo ls
NAME            URL
kx-insights     https://portal.dl.kx.com/assets/helm

  2. If the appropriate repo is not available, you can obtain access as follows: 

    # The credentials used to create the `docker-registry` secret above can be used here.
helm repo add kx-insights https://portal.dl.kx.com/assets/helm --username **** --password ****

  3. You can now search for the chart and determine the appropriate chart and app version 

    helm search repo kx-insights/kxi-bundle
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
kx-insights/kxi-bundle          1.2.3           1.2.3           A Helm chart for Kubernetes

  4. In order to download the charts, as well as untar them into your local session, you can run the following: 

    helm fetch kx-insights/kxi-bundle --version  1.2.3 --untar

Launch

Running these commands starts the kdb Insights Database, a Reliable Transport (RT stream), a publisher to the RT stream and a monitoring chart. The database includes sample schemas in the kxi-db/assembly.yaml configuration file. If you need to introduce additional schemas, add them to this configuration file.

The kxi-bundle chart is a wrapper to the following helm charts.

  1. kxi-rt-q-pub, a publisher to the RT stream
  2. kxi-rt, a RT stream called kxi-mystream
  3. kxi-db, a database subscribing to the stream kxi-mystream
  4. kxi-monitoring, a prometheus and grafana pod to monitor the data flow

Subscription

The kdb Insights database has been configured to subscribe to the RT stream deployed. There is a table schema for a trade table that is populated with the data published by the kxi-rt-q-pub chart.

helm install kxi kxi-bundle -f kxi-bundle/global_settings.yaml

Query

The following examples demonstrate how you can query data using kdb Insights through its interfaces.

To query data, you first need to port-forward the Gateway port running on port 5050 (tcp/ip) or 8080 (http):

kubectl port-forward service/kxi-gw 5050
kubectl port-forward service/kxi-gw 8080

To query data using REST, make a GET request with the following: 

curl -X GET -H 'Content-Type: application/json' http://localhost:8080/data -d '{"table":"trade"}'

To query data using q-based queries, use the following: 

q)h:hopen 5050  // the Gateway has a tcp port forwarded locally on this port
q)h(`.kxi.getData;enlist[`table]!enlist`trade;`;()!())
q)h(`.kxi.qsql; enlist[`query]!enlist"select from trade";`;()!())
q)h(`.kxi.sql; enlist[`query]!enlist"select * from trade";`;()!())

!! note "The pre-requisites to getting the kxi-python interface installed can be found here".

Connect to your instance of kdb Insights and query using sql returning result as json payload. 

>>> import kxi.query
>>> conn = kxi.query.Query('http://localhost:8080', usage='MICROSERVICES', data_format='application/json')
>>> conn.sql('SELECT * FROM trade')

To query data using the kdb Insights Command Line Interface (CLI), run the following command: 

kxi query --sql "SELECT * FROM taxi"

For further details on how to configure the CLI, see the kdb Insights Command Line Interface overview.

Publish

The following examples demonstrate how you can publish data using kdb Insights through its interfaces.

To publish data using these examples, you will first have to port-forward the RT replicator endpoint. 

kubectl port-forward service/kxi-mystream-0 5002

To publish data using q, run the following commands:

Download the rt.qpk, which provides APIs for publishing and subscribing to an RT stream.

version=1.13.0
curl -L  https://portal.dl.kx.com/assets/raw/rt/$version/rt.$version.qpk -o rt.$version.qpk
unzip rt.$version.qpk

Start a q session and publish the data using the RT publish API. 

$ cd rt/
$ q startq.q
q)params:(`path`stream`publisher_id`cluster)!("/tmp/rt";"data";"pub1";enlist(":127.0.0.1:5002"))
q)p:.rt.pub params;
q)show data:enlist`time`sym`exch`side`price`size`tradeID!(.z.p;`KX;`LSE;`buy;10.;100;first -1?0Ng);
q)p(`.b; `trade; data)

!!! note "This assumes you have a trade.csv file with data that matches the schema of the trade table.

To publish data you can then run the following command: 

export KXI_C_SDK_APP_LOG_PATH=$(pwd)  ## useful for debug logging, defaults to /tmp
kxi publish --mode rt --file-format csv --table trade --data config/trade.csv --endpoint :localhost:5002

Refer to the kdb Insights Command Line Interface documentation for details on configuring the CLI.

To publish data using Python, refer to the kxi-python documentation.

Monitor

A helm chart has been provided, which includes metrics illustrating the data flow of data into the kdb Insights database from RT. The chart makes use of prometheus and grafana to plot the data being sent to the system.

To access the dashboard, the grafana service should be port forwarded:

kubectl port-forward service/grafana 3000
Metric Description
RT messages sent by publisher The number of messages published by each publisher to RT
RT messages merged by publisher After receiving data from a publisher, RT must merge it before it is sent to subscribers
Total messages merged The sum of messages from all publisher merged by RT. This should match 'RT messages merged by publisher'.
Total messages sequenced The sum of messages from all publisher received by RT
DAP messages received from RT The number of messages (payload) and records (row count) received by the Data Access Process (DAP) from RT
SM messages received from RT The number of messages (payload) and records (row count) received by the Storage Manager (SM) from RT