Skip to content

Quickstart

Introduction

This quickstart page guides you through streaming data from kdb Insights Enterprise over WebSockets. If you are using the kdb Insights SDK, refer to the WebSockets guide.

For more details on how the protocol works, refer to the protocol documentation. All examples below use JSON. For information on using QIPC instead, refer to the protocol documentation.

Prerequisites

  • You have access to a running version of kdb Insights Enterprise. For more information, refer to the installation guide.
  • You have installed the kdb Insights Command Line following the installation guide.

For the following example, assume you have an assembly called sdk-sample-assembly with a pipeline called sdtransform.

Quickstart

Authentication and Authorization

Before pushing a package or publishing data, use the KXI CLI to create users, clients, and assign roles. For more information on the CLI, refer to the kdb Insights Command Line documentation.

Administrator password

The administrator password below is defined during the installation of your kdb Insights Enterprise deployment. For more information, refer to the Administration Passwords documentation.

# Create a user
kxi user create user@domain.com --password <myPassword> --admin-password <adminPassword> --not-temporary
INSIGHTS_ROLES="insights.role.maintainer,insights.client.create,insights.client.delete,insights.package.install,insights.package.delete,insights.admin.package.system,insights.package.list,insights.package.update,insights.package.upload,insights.query.stream"
kxi user assign-roles user@domain.com --roles $INSIGHTS_ROLES --admin-password <adminPassword>

## Create a client
CLIENT_ID=svc1
INSIGHTS_ROLES="insights.client.create,insights.client.delete,insights.package.install,insights.package.delete,insights.admin.package.system,insights.package.list,insights.package.update,insights.package.upload,insights.role.developer,insights.query.stream"
kxi user create-client $CLIENT_ID --admin-password <adminPassword>
kxi user assign-roles service-account-$CLIENT_ID --roles $INSIGHTS_ROLES --admin-password <adminPassword>
CLIENT_SECRET=$(kxi user get-client-secret $CLIENT_ID --admin-password <adminPassword>)

Package Deployment

Follow the steps below to download and deploy the sample package kxi-ws-1.11.0.kxi, available from the KX Download Portal.

Step 1 - Download the package

Run the following command to download the package:

version=1.11.0
curl -L https://portal.dl.kx.com/assets/raw/package-samples/kxi-ws/$version/kxi-ws-$version.kxi -o kxi-ws-$version.kxi

Step 2 - Push the package to kdb Insights

Push the downloaded package to kdb Insights using the following command:

kxi package push kxi-ws-1.11.0.kxi --client-id $CLIENT_ID --client-secret $CLIENT_SECRET

Step 3 - Deploy the package

Deploy the package to kdb Insights:

kxi package deploy kxi-ws/1.11.0 --client-id $CLIENT_ID --client-secret $CLIENT_SECRET

WebSocket connection and subscription

Step 1 - Establish connection

Use the Linux utility wscat to establish a WebSocket client connection.

PACKAGE_NAME=kxi-ws
PIPELINE_NAME=wsstream
HOSTNAME=insights.kx.com  ## replace with your own hostname
URL=https://$HOSTNAME/servicegateway/ws/v1/subscribe/${PACKAGE_NAME}-${PIPELINE_NAME}
# Login to kx insights with a serivce account
kxi  --serviceaccount-id=${CLIENT_ID} --serviceaccount-secret=${CLIENT_SECRET} auth login --serviceaccount
# Obtain the access token
TOKEN=$(kxi auth print-token)
# Make the websockets connection
wscat -H "Sec-WebSocket-Protocol: Bearer $TOKEN"   -c $URL

Step 2 - Send subscription

There are three ways to subscribe to data over the WebSocket client. You can refer to the examples of each method below. For more information on subscription requests, refer to the client protocol page.

To get a single snapshot of the current data in the table "trace", set the type to snap: 

 {"type":"snap","payload":{"topic":"trace"},"id":1}
You'll receive the following response: 
{"payload":{"data":{"readTS":["2021-01-01 00:00:00.0","2021-01-01 00:00:01.000000001","2021-01-01 00:00:02.0","2021-01-01 00:00:03.0","2021-01-01 00:00:04.0","2021-01-01 00:00:05.0","2021-01-01 00:00:06.0","2021-01-01 00:00:07.0","2021-01-01 00:00:08.0","2021-01-01 00:00:09.0"],"valFloat":[72.0,81.0,81.0,90.0,34.0,50.0,51.0,46.0,45.0,59.0],"alarm":[0,0,1,0,1,0,0,1,1,0],"qual":[1,1,1,0,0,1,1,0,1,0],"updateTS":["2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492"],"captureTS":["2021-01-01 00:00:00.0","2021-01-01 00:00:01.000000001","2021-01-01 00:00:02.0","2021-01-01 00:00:03.0","2021-01-01 00:00:04.0","2021-01-01 00:00:05.0","2021-01-01 00:00:06.0","2021-01-01 00:00:07.0","2021-01-01 00:00:08.0","2021-01-01 00:00:09.0"],"sensorID":[1,1,1,1,1,1,1,1,1,1]}},"id":1,"type":"snapped"}

To subscribe to incremental refreshes of the data without receiving a snapshot, set the type to subscribe:

{"type":"subscribe", "id":2, "payload":{"topic":"trace"}}
You'll receive a "subscribed" response, followed by the "update" messages: 
{"payload":{"subscription":"3f108d8c-b84b-4b2c-a13c-7906ff3037b5"},"id":2,"type":"subscribed"}
{"payload":{"data":{"readTS":["2021-01-01 00:00:00.0"],"valFloat":[72.0],"alarm":[0],"qual":[1],"updateTS":["2024-08-07 13:52:03.948521052"],"captureTS":["2021-01-01 00:00:00.0"],"sensorID":[1]},"subscription":"3f108d8c-b84b-4b2c-a13c-7906ff3037b5"},"id":2,"type":"update"}

The subsnap message type combines both the snap and subscribe messages. You'll receive a snapshot of the data, followed by incremental updates: 

{"type":"subsnap", "id":3, "payload":{"topic":"trace"}}
After sending this, you'll receive a snapshot of the data, followed by the updates: 
{"payload":{"data":{"readTS":["2021-01-01 00:00:00.0","2021-01-01 00:00:01.000000001","2021-01-01 00:00:02.0","2021-01-01 00:00:03.0","2021-01-01 00:00:04.0","2021-01-01 00:00:05.0","2021-01-01 00:00:06.0","2021-01-01 00:00:07.0","2021-01-01 00:00:08.0","2021-01-01 00:00:09.0"],"valFloat":[72.0,81.0,81.0,90.0,34.0,50.0,51.0,46.0,45.0,59.0],"alarm":[0,0,1,0,1,0,0,1,1,0],"qual":[1,1,1,0,0,1,1,0,1,0],"updateTS":["2024-08-07 13:52:03.948521052","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492"],"captureTS":["2021-01-01 00:00:00.0","2021-01-01 00:00:01.000000001","2021-01-01 00:00:02.0","2021-01-01 00:00:03.0","2021-01-01 00:00:04.0","2021-01-01 00:00:05.0","2021-01-01 00:00:06.0","2021-01-01 00:00:07.0","2021-01-01 00:00:08.0","2021-01-01 00:00:09.0"],"sensorID":[1,1,1,1,1,1,1,1,1,1]},"subscription":"187ea364-af48-4d83-9f2a-a0db55ab6de8"},"id":3,"type":"subsnapped"}
{"payload":{"data":{"readTS":["2021-01-01 00:00:00.0"],"valFloat":[72.0],"alarm":[0],"qual":[1],"updateTS":["2024-08-07 13:53:26.878540188"],"captureTS":["2021-01-01 00:00:00.0"],"sensorID":[1]},"subscription":"187ea364-af48-4d83-9f2a-a0db55ab6de8"},"id":3,"type":"update"}

To unsubscribe from a subscription or subsnap, send an unsubscribe message using the subscription UUID: 

{"type":"unsubscribe", "id":4, "payload":{"subscription":"187ea364-af48-4d83-9f2a-a0db55ab6de8"}}
You'll receive the following response: 
{"payload":{"subscription":"187ea364-af48-4d83-9f2a-a0db55ab6de8"},"id":4,"type":"unsubscribed"}

Message ID

The ID in the above messages is initially set by the client. Any messages sent from the Service Gateway to the client include the ID of the original message they are responding to. This number must be unique for every message sent from a given client to the Service Gateway and must be incremented with each message sent.