Quickstart
Introduction
This quickstart page guides you through streaming data from kdb Insights Enterprise over WebSockets. If you are using the kdb Insights SDK, refer to the WebSockets guide.
For more details on how the protocol works, refer to the protocol documentation. All examples below use JSON. For information on using QIPC instead, refer to the protocol documentation.
Prerequisites
- You have access to a running version of kdb Insights Enterprise. For more information, refer to the installation guide.
- You have installed the kdb Insights Command Line following the installation guide.
For the following example, assume you have an assembly called sdk-sample-assembly
with a pipeline called sdtransform
.
Quickstart
Authentication and Authorization
Before pushing a package or publishing data, use the KXI CLI to create users, clients, and assign roles. For more information on the CLI, refer to the kdb Insights Command Line documentation.
Administrator password
The administrator password below is defined during the installation of your kdb Insights Enterprise deployment. For more information, refer to the Administration Passwords documentation.
# Create a user
kxi user create user@domain.com --password <myPassword> --admin-password <adminPassword> --not-temporary
INSIGHTS_ROLES="insights.role.maintainer,insights.client.create,insights.client.delete,insights.package.install,insights.package.delete,insights.admin.package.system,insights.package.list,insights.package.update,insights.package.upload,insights.query.stream"
kxi user assign-roles user@domain.com --roles $INSIGHTS_ROLES --admin-password <adminPassword>
## Create a client
CLIENT_ID=svc1
INSIGHTS_ROLES="insights.client.create,insights.client.delete,insights.package.install,insights.package.delete,insights.admin.package.system,insights.package.list,insights.package.update,insights.package.upload,insights.role.developer,insights.query.stream"
kxi user create-client $CLIENT_ID --admin-password <adminPassword>
kxi user assign-roles service-account-$CLIENT_ID --roles $INSIGHTS_ROLES --admin-password <adminPassword>
CLIENT_SECRET=$(kxi user get-client-secret $CLIENT_ID --admin-password <adminPassword>)
Package Deployment
Follow the steps below to download and deploy the sample package kxi-ws-1.11.0.kxi
, available from the KX Download Portal.
Step 1 - Download the package
Run the following command to download the package:
version=1.11.0
curl -L https://portal.dl.kx.com/assets/raw/package-samples/kxi-ws/$version/kxi-ws-$version.kxi -o kxi-ws-$version.kxi
Step 2 - Push the package to kdb Insights
Push the downloaded package to kdb Insights using the following command:
kxi package push kxi-ws-1.11.0.kxi --client-id $CLIENT_ID --client-secret $CLIENT_SECRET
Step 3 - Deploy the package
Deploy the package to kdb Insights:
kxi package deploy kxi-ws/1.11.0 --client-id $CLIENT_ID --client-secret $CLIENT_SECRET
WebSocket connection and subscription
Step 1 - Establish connection
Use the Linux utility wscat
to establish a WebSocket client connection.
PACKAGE_NAME=kxi-ws
PIPELINE_NAME=wsstream
HOSTNAME=insights.kx.com ## replace with your own hostname
URL=https://$HOSTNAME/servicegateway/ws/v1/subscribe/${PACKAGE_NAME}-${PIPELINE_NAME}
# Login to kx insights with a serivce account
kxi --serviceaccount-id=${CLIENT_ID} --serviceaccount-secret=${CLIENT_SECRET} auth login --serviceaccount
# Obtain the access token
TOKEN=$(kxi auth print-token)
# Make the websockets connection
wscat -H "Sec-WebSocket-Protocol: Bearer $TOKEN" -c $URL
Step 2 - Send subscription
There are three ways to subscribe to data over the WebSocket client. You can refer to the examples of each method below. For more information on subscription requests, refer to the client protocol page.
To get a single snapshot of the current data in the table "trace"
, set the type
to snap
:
{"type":"snap","payload":{"topic":"trace"},"id":1}
{"payload":{"data":{"readTS":["2021-01-01 00:00:00.0","2021-01-01 00:00:01.000000001","2021-01-01 00:00:02.0","2021-01-01 00:00:03.0","2021-01-01 00:00:04.0","2021-01-01 00:00:05.0","2021-01-01 00:00:06.0","2021-01-01 00:00:07.0","2021-01-01 00:00:08.0","2021-01-01 00:00:09.0"],"valFloat":[72.0,81.0,81.0,90.0,34.0,50.0,51.0,46.0,45.0,59.0],"alarm":[0,0,1,0,1,0,0,1,1,0],"qual":[1,1,1,0,0,1,1,0,1,0],"updateTS":["2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492"],"captureTS":["2021-01-01 00:00:00.0","2021-01-01 00:00:01.000000001","2021-01-01 00:00:02.0","2021-01-01 00:00:03.0","2021-01-01 00:00:04.0","2021-01-01 00:00:05.0","2021-01-01 00:00:06.0","2021-01-01 00:00:07.0","2021-01-01 00:00:08.0","2021-01-01 00:00:09.0"],"sensorID":[1,1,1,1,1,1,1,1,1,1]}},"id":1,"type":"snapped"}
To subscribe to incremental refreshes of the data without receiving a snapshot, set the type
to subscribe
:
{"type":"subscribe", "id":2, "payload":{"topic":"trace"}}
{"payload":{"subscription":"3f108d8c-b84b-4b2c-a13c-7906ff3037b5"},"id":2,"type":"subscribed"}
{"payload":{"data":{"readTS":["2021-01-01 00:00:00.0"],"valFloat":[72.0],"alarm":[0],"qual":[1],"updateTS":["2024-08-07 13:52:03.948521052"],"captureTS":["2021-01-01 00:00:00.0"],"sensorID":[1]},"subscription":"3f108d8c-b84b-4b2c-a13c-7906ff3037b5"},"id":2,"type":"update"}
The subsnap
message type combines both the snap
and subscribe
messages. You'll receive a snapshot of the data, followed by incremental updates:
{"type":"subsnap", "id":3, "payload":{"topic":"trace"}}
{"payload":{"data":{"readTS":["2021-01-01 00:00:00.0","2021-01-01 00:00:01.000000001","2021-01-01 00:00:02.0","2021-01-01 00:00:03.0","2021-01-01 00:00:04.0","2021-01-01 00:00:05.0","2021-01-01 00:00:06.0","2021-01-01 00:00:07.0","2021-01-01 00:00:08.0","2021-01-01 00:00:09.0"],"valFloat":[72.0,81.0,81.0,90.0,34.0,50.0,51.0,46.0,45.0,59.0],"alarm":[0,0,1,0,1,0,0,1,1,0],"qual":[1,1,1,0,0,1,1,0,1,0],"updateTS":["2024-08-07 13:52:03.948521052","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492","2024-08-07 13:30:32.997582492"],"captureTS":["2021-01-01 00:00:00.0","2021-01-01 00:00:01.000000001","2021-01-01 00:00:02.0","2021-01-01 00:00:03.0","2021-01-01 00:00:04.0","2021-01-01 00:00:05.0","2021-01-01 00:00:06.0","2021-01-01 00:00:07.0","2021-01-01 00:00:08.0","2021-01-01 00:00:09.0"],"sensorID":[1,1,1,1,1,1,1,1,1,1]},"subscription":"187ea364-af48-4d83-9f2a-a0db55ab6de8"},"id":3,"type":"subsnapped"}
{"payload":{"data":{"readTS":["2021-01-01 00:00:00.0"],"valFloat":[72.0],"alarm":[0],"qual":[1],"updateTS":["2024-08-07 13:53:26.878540188"],"captureTS":["2021-01-01 00:00:00.0"],"sensorID":[1]},"subscription":"187ea364-af48-4d83-9f2a-a0db55ab6de8"},"id":3,"type":"update"}
To unsubscribe from a subscription or subsnap, send an unsubscribe message using the subscription UUID:
{"type":"unsubscribe", "id":4, "payload":{"subscription":"187ea364-af48-4d83-9f2a-a0db55ab6de8"}}
{"payload":{"subscription":"187ea364-af48-4d83-9f2a-a0db55ab6de8"},"id":4,"type":"unsubscribed"}
Message ID
The ID in the above messages is initially set by the client. Any messages sent from the Service Gateway to the client include the ID of the original message they are responding to. This number must be unique for every message sent from a given client to the Service Gateway and must be incremented with each message sent.