Skip to content

Publishing and subscribing to kdb Insights Enterprise using q

In this example we will work through a q process sending data to an instance of kdb Insights Enterprise using a 1 or 3 node deployment of RT. The publisher will be running outside of the kdb Insights Enterprise deployment, meaning you could be sending data to kdb Insights Enterprise using a q session running on your laptop. You can use the same q process to subscribe to the data published.

Reliable Transport (RT) comes with a set of q functions to send or receive message to and from RT. Details on these q function are touched on below, but for more information see here

Authentication

Before data can be sent to kdb Insights Enterprise, a set of authentication steps need to be followed.

These steps are outlined on the kdb Insights Enterprise client enrolment page.

Interface

The q interface to RT comes with a number of APIs including publish and subscribe APIs.

In order to access the APIs, download the rt.qpk.

version=1.14.0
curl -L  https://portal.dl.kx.com/assets/raw/rt/$version/rt.$version.qpk -o rt.qpk
unzip rt.qpk

Publish

In this example you can use the RT publish function, .rt.pub. This is an API, maintained by KX, that can be called to send data to a RT stream.

Start a q session and publish the data 

$ cd rt
$ q startq.q
q)
q)setenv[`KXI_CONFIG_URL] "https://my-deploy.com/informationservice/details/d955a31fc4deb880e53704794671bd49"; // Note, this is taken from the client enrolment step above
q)params:`config_url`path!(getenv`KXI_CONFIG_URL; "/tmp/rt/");
q)params
config_url| "https://my-deploy.com/informationservice/details/d955a31fc4deb880e53704794671bd49"  
path      | "/tmp/rt/"               // location where the user wishes to write the RT log file. A location with suitable disk space should be chosen.
q)p:.rt.pub[params];            // initialize your connection to RT. This creates a foreign object used to publish.
q)data:([] col1:`a`b`c; col2:1 2 3)
q)p(`.b; `table; data)  // publish to RT

Data format

RT is data format agnostic. The 3 item list we are publishing above is a messageType, table name and payload. The kdb Insights Database and Stream Processor expect data in this format.

Subscribe

In this example you can use the RT subscribe function, .rt.sub. This is an API, maintained by KX, that can be called to subscribe to data from a RT stream.

Start a q session and subscribe to the data 

$ cd rt
$ q startq.q
q)
q)setenv[`KXI_CONFIG_URL] "https://my-deploy.com/informationservice/details/d955a31fc4deb880e53704794671bd49"; // Note, this is taken from the client enrolment step above
q)upd:{[msg;pos] show .debug.upd:`msg`pos!(msg;pos)}  // callback function executed when a new message is received
q)pos:0; // position in the RT stream to subscribe from
q)filter:"";  // populate with a table to receive a subset of the data. when left blank receive all data
q)params:`config_url`position`callback`filter!(getenv`KXI_CONFIG_URL;pos;upd;filter );
q)params
config_url| "https://my-deploy.com/informationservice/details/d955a31fc4deb880e53704794671bd49"  
callback  | {[msg;pos]show .debug.upd:`msg`pos!(msg;pos)}
position  | 0
filter    | ""
q)
q)s:.rt.sub params;            // initialize your connection to RT. This creates a foreign object. To close the connection, s:()
q)msg| (`.b;`table;+`col1`col2!(`a`b`c;1 2 3))
pos| 228698418577408

Querying Data

You have now sent data into kdb Insights Enterprise. There are a number of ways of querying that data back, including the REST call getData and SQL.