Publishing to kdb Insights Enterprise using q
In this example we will work through a q process sending data to an instance of kdb Insights Enterprise using a 1 or 3 node deployment of RT. The publisher will be running outside of the kdb Insights Enterprise deployment, meaning you could be sending data to kdb Insights Enterprise using a q session running on your laptop.
Reliable Transport (RT) comes with a set of q functions to send or receive message to and from RT. Details on these q function are touched on below, but for more information see here
Authentication
Before data can be sent to kdb Insights Enterprise, a set of authentication steps need to be followed. These steps are outlined below.
The steps below show you how to authenticate and enrol a publisher client to RT, using the KXI CLI.
Create a user
The steps below assume that the user executing the steps has been provided with a set of credentials to log in to their kdb Insights Enterprise deployment.
- Configure your KXI CLI with the details of your deployment and namespace
$ kxi configure Profile type (enterprise, microservices) [enterprise]: Hostname []: https://my-deploy.com/ Namespace []: namespace Service account ID []: Encryption enabled (true/false) [true]: CLI successfully configured, configuration stored in /home/user/.insights/cli-config
The file created should look like
$ cat ~/.insights/cli-config
[default]
usage = enterprise
hostname = https://my-deploy.com/
namespace = namespace
auth.serviceaccount.id =
auth.serviceaccount.secret =
encryption = true
admin.password = password
admin.password
The entry for admin.password
must be manually added to the file above. It will not be added using the kxi configure
command.
- Authenticate against your insights deploy You should login with the credentials your Insights administrator have provided, for example user@company.com
kxi auth login
-
Create a client in keycloak
kxi user create-client my-publisher
-
Assign the relevant permissions to that client
kxi user assign-roles service-account-my-publisher --roles insights.role.maintainer
-
Obtain the client secret for the client
kxi user get-client-secret my-publisher
-
Configure your KXI CLI with the details obtained for the client ID and client secret
kxi configure Profile type (enterprise, microservices) [enterprise]: enterprise Hostname [https://my-deploy.com/]: Namespace [namespace]: namespace Service account ID [my-app]: my-publisher Service account Secret (input hidden): Re-enter to confirm (input hidden): Encryption enabled (true/false) [false]: CLI successfully configured, configuration stored in /home/username/.insights/cli-config
-
Identify your external reference ID If using an assembly file, this will be the value "ext-taxi"
sequencer: transport: external: true topicConfig: subTopic: "ext-taxi"
-
Enrol a client using the kxi cli
$ kxi client enrol --name my-publisher --insert-topic ext-taxi { "message": "success", "detail": "Client enrolled", "url": "d955a31fc4deb880e53704794671bd49", "config_url": "https://my-deploy.com/informationservice/details/d955a31fc4deb880e53704794671bd49" }
Publish
In this example, we will make use of a RT publish function, .rt.pub
. This is an API, maintained by KX, that can be called to send data into RT.
In order to access the API, download the rt.qpk
.
version=1.10.0
curl -L https://portal.dl.kx.com/assets/raw/rt/$version/rt-$version.qpk -o rt.tar.gz
unzip rt.tar.gz
Start a q session and publish the data
$ cd rt
$ q startq.q
q)
q)setenv[`KXI_CONFIG_URL] "https://my-deploy.com/informationservice/details/d955a31fc4deb880e53704794671bd49"; // Note, this is taken from the client enrolment step above
q)params:`config_url`path!(getenv`KXI_CONFIG_URL; "/tmp/rt/");
q)params
config_url| "https://my-deploy.com/informationservice/details/d955a31fc4deb880e53704794671bd49"
path | "/tmp/rt/" // location where the user wishes to write the RT log file. A location with suitable disk space should be chosen.
q)p:.rt.pub[params]; // initialize your connection to RT. This creates a foreign object used to publish.
q)data:([] col1:`a`b`c; col2:1 2 3)
q)p(`.b; `table; data) // publish to RT
Data format
RT is data format agnostic. The 3 item list we are publishing above is a messageType, table name and payload. The kdb Insights Database and Stream Processor expect data in this format.
Querying Data
You have now sent data into kdb Insights Enterprise. There are a number of ways of querying that data back, including the REST call getData and SQL.