Skip to content

Publishing to kdb Insights Enterprise using q

In this example we will work through a q process sending data to an instance of kdb Insights Enterprise using a 1 or 3 node deployment of RT. The publisher will be running outside of the kdb Insights Enterprise deployment, meaning you could be sending data to kdb Insights Enterprise using a q session running on your laptop.

Reliable Transport (RT) comes with a set of q functions to send or receive message to and from RT. Details on these q function are touched on below, but for more information see here

Authentication

Before data can be sent to kdb Insights Enterprise, a set of authentication steps need to be followed. These steps are captured here. The instructions provide details on how to obtain a KXI_CONFIG_URL. This URL acts as a discovery service for RT and is only made available if the authentication steps are completed successfully.

QPacker

QPacker is a packaging utility which we will use here to create an image, known as a QPK, that will contains a set of q functions.

Installation

Installation details for QPacker can be found here

Create a q script

In this example, we will make use of a RT publish function, .rt.pub. This is an API, maintained by KX, that can be called to send data into RT.

q)params:`config_url`path!(getenv`KXI_CONFIG_URL; "/tmp/rt/");
q)params
config_url| ""                       // obtained as part of the authenction steps above, corresponds to a URL which will return the location of the RT replicators.
path      | "/tmp/rt/"               // location where the user wishes to write the RT log file. A location with suitable disk space should be chosen.   
.rt.push:.rt.pub[params];            // .rt.push becomes a projection which can be called anytime the users wishes to send data to RT.

The text below can be pasted into a q file, called pub.q

// pub.q
params:`config_url`path!(getenv`KXI_CONFIG_URL; "/tmp/rt/");

init:{[params]
  .rt.push:.rt.pub params;  
  }

.sample.getEmail:{enlist`host`time`org`activity!(`fd.com;.z.p;`KX;`open)}  // random data about an email marketing campaign;
.sample.getAccounts:{ungroup enlist`org`countryHQ`orgWebsite!((`FD;`KX);(`UK;`US);(`fd.com;`kx.com)};  // random account data

.sample.pubEmail:{
  .rt.push (`.b; `email;.sample.getEmail[])
  }
.sample.pubAccounts:{
   .rt.push (`.b; `accounts;  .sample.getAccounts[])
  }

Message format

RT is data format agnostic, it can send and receive any form of data. However, in this case we are relying on a message format which aligns to the message format that a downstream subscriber, kdb Insights Enterprise, is expecting from RT.

Consider the function to publish email data to RT defined in pub.q above. There is a 3 item list passed to the projection .rt.push:

variable description
.b This is a “bulk data” record, similar in spirit to kdb tick’s upd message, but using a name not in the user namespace
email The table name being sent
payload The data to be stored in the table

Download the rt.qpk

This can be provided by the KX support team. Details on the rt.qpk are available here

Build a QPK

Extensive details on how to build a QPK can be found here

A brief example is provided below:

Note

You should have Docker Desktop running in advance of this step.

$ ls -al
total 29552
drwxr-xr-x  2 user user     4096 Jun  9 14:56 .
drwxr-xr-x 14 user user     4096 May 29 15:59 ..
-rwxr-xr-x  1 user user      414 Jun  8 15:08 pub.q  ## created in the step prior
-rwxr-xr-x  1 user user      127 May 29 16:01 qp.json
-rw-r--r--  1 user user       58 May 29 15:59 qp_docker_include_file
-rw-r--r--  1 user user 30230736 Jun  8 11:41 rt.qpk

$ cat qp_docker_include_file
RUN yum -y update && yum -y install procps net-tools lsof

$ cat qp.json
{
  "q-pub": {
    "entry": [ "pub.q" ],
    "depends": [ "rt" ],   // this is where the rt.qpk is included in your QPK
    "docker_base_image": [ "rockylinux/rockylinux:8" ]
  }
}

$ qp build  ## this will build a qpack called q-pub, based on the contents of qp.json

Start the QPK

Note

You will need to have a valid q license present in $HOME/.qp.licenses/kc.lic.

Tip

If you wish to start a 1 node RT publisher, you can add an env variable to your qpbuild/.env file: 

RT_REPLICAS=1
Details on the different RT node sizes are available here

The following command will start a docker container with your q code now available:

$ qp run q-pub
INFO  | License | Using kdb+ license file [/home/user/.qp.licenses/kc.lic]
INFO  | Run     | Target [q-pub]
INFO: Parse kdb+ license [kc.lic] from KDB_LICENSE_B64 environment variable
KDB+ 4.0 Cloud Edition 2023.01.20 Copyright (C) 1993-2023 Kx Systems
l64/ 8()core 25377MB nobody a7599d8e10fa 172.17.0.3 EXPIRE 2024.01.13 user@kx.com KXMS #76655

q)
q).sample
           | ::
getEmail   | locked
getAccounts| locked
pubEmail   | locked
pubAccounts| locked
q)

Publishing data

A sample set of q commands to publish data to kdb Insights Enterprise via RT:

q)params
config_url| "https://user.aws-green.kxi-dev.kx.com/informationservice/detai...
path      | "/tmp/rt/"
q)
q)pub_function:.rt.pub params // this will do a few things, include starting the RT replicators and define yourself as a publisher to RT
q)
q) // generate some random data, note this could come from a csv etc
q)show sample:ungroup enlist`org`countryHQ`orgWebsite!((`FD;`KX);(`UK;`US);(`fd.com;`kx.com))
org countryHQ orgWebsite
------------------------
FD  UK        fd.com
KX  US        kx.com
q)
q)// sent that data to insights via RT.
q)pub_function (`.b; `accounts; sample)
158

Querying Data

You have now sent data into kdb Insights Enterprise. There are a number of ways of querying that data back, including the REST call getData and SQL.