Skip to content

Running RT outside of a container

This page demonstrates how to run the RT service outside of a container. It also includes examples to publish and subscribe to an RT stream using q.

Prerequisites

  1. A kdb+ license. If you do not have a license, obtain a trial license on the KX download site.

Download

Step 1 - Obtain a bearer token

  1. Log into the KX downloads portal.
  2. After authentication, navigate to the portal authentication page.
  3. Click Add New Bearer Token.
  4. Click Copy Bearer.

Step 2 - Download the artefacts

Download the kxi-rt.qpk:

version=1.12.0
USERID=<userid for the kx downloads portal>
BEARER=<bearer token>
curl -L  --user $USERID:$BEARER https://portal.dl.kx.com/assets/raw/kxi-rt/$version/kxi-rt.$version.qpk -o kxi-rt.qpk

Start

Step 1 - Install kdb+

You need a kdb+ license to run RT. Follow the install guide to obtain your license and set up your environment.

QHOME=~/q/

Your $QHOME directory should resemble the following structure:

├── kc.lic
├── l64
│   └── q
└── q.k

Step 2 - Start the RT service

A shell script is provided to start your RT stream:

RT Port Usage

RT uses multiple ports. Before starting RT, identify an available port range. RT defines a base port, and the required ports are incremented from this base. The default range includes ports 6000 to 6016.

unzip kxi-rt.qpk
cd kxi-rt 
./start_rt.sh

When you run ./start_rt.sh, default values are used. You can override these defaults by passing in alternative values:

"Usage: start_rt.sh [-p port] [-i in_dir] [-o out_dir] [-t time] [-d disk] [-l limit]"
argument default description
p 6000 Base port where RT starts. Ports used increment from this base.
i /tmp/s/in Directory where RT stores input logs.
o /tmp/s/out Directory where RT stores output logs. Logs move from the input directory to the output directory after RAFT consensus.
t 1440 Retention period for merged log files in minutes. Refer to the RT archival documentation for more information.
d 90 Maximum percentage of disk space used by RT. Refer to the RT archival documentation for more information.
l 10Gi Maximum size of all log files in RT. Refer to the RT archival documentation for more information.

RT clients

You can use any of the RT interfaces to publish data to your RT stream.

To use the q API for publishing data, you must download the rt.qpk:

version=1.12.0
USERID=<userid for the kx downloads portal>
BEARER=<bearer token>
curl -L --user $USERID:$BEARER  https://portal.dl.kx.com/assets/raw/rt/$version/rt.$version.qpk -o rt.qpk

RT stream connection

The following publisher and subscriber clients connect to RT using the cluster key. This is the endpoint where the RT stream replicators run. There are two server side replicators, push_server for publishers and pull_server for subscribers. With a default RT base port of 6000, these replicators run on ports 6016 and 6015.

Publish

You can use rt.qpk to publish to the RT stream. Start a q session to publish the data: 

$ unzip rt.qpk
$ cd rt
$ q startq.q
q)
q)params:(`path`stream`cluster`publisher_id)!("/tmp/rt_pub";"data";enlist":localhost:6016";"mypub");
q)p:.rt.pub params;            // initialize your connection to RT. This creates a foreign object, that can be used to publish your data.
q)tab:([] col1:`a`b`c; col2:1 2 3)
q)p(`upd; `tab; tab)  // publish to RT

Data format

RT is data format agnostic. However, to enable future table filtering, publish a three item list. The list must contain the message type, table name, and payload.

Subscribe

You can also use the rt.qpk to subscribe to the RT stream:

$ q startq.q
q)
q)tab:([] col1:`$(); col2:`long$());
q)position:0;
q)upd:{[msg;pos] show `msg`pos!(msg;pos); if[(t:msg 1)in tables[]; upsert[t; msg 2]];}
q)params:`path`cluster`stream`position`callback!("/tmp/rt_sub"; enlist":localhost:6015"; "data"; position; upd);
q).rt.sub params;

Payload from RT

When a client subscribes to an RT stream, they define a callback function. This function is triggered whenever RT sends a new message to the subscriber. It accepts two parameters, the first parameter is the message from the publisher, and the second is the position in the RT stream. This position allows subscribers to cache it, and if they restart, resubscribe from the cached position to ensure no data is missed from the RT.