Using the kdb Insights Enterprise SDKs

This use case highlights how to deploy the sdk-sample-assembly Custom Resource (CR) pipeline, publish data into it and query the data, all through secure and authenticated endpoints within kdb Insights Enterprise.

The goal is to take the data pipeline highlighted below and deploy to the kdb Insights Enterprise Base Application to allow data to be ingested, persisted and queried.

Deploying the Assembly CR

You can deploy the assembly on top of the base instance of the kdb Insights Enterprise in your cluster.

To deploy the assembly, download the sdk_sample_assembly.yaml from the release repository and deploy to your cluster with the command.

kxi assembly deploy --filepath sdk_sample_assembly.yaml

The label insights.kx.com/app currently gets set to the name of the assembly so this can be used as a filter to view the resources. For example if you deployed an assembly called sdk-sample-assembly you can see the pods associated with the assembly by running

$ kubectl get pods -l insights.kx.com/app=sdk-sample-assembly
NAME                                   READY          STATUS          RESTARTS    AGE
sdk-sample-assembly-dap-hdb-0                       2/2     Running   0          2d15h
sdk-sample-assembly-dap-hdb-1                       2/2     Running   0          2d15h
sdk-sample-assembly-dap-hdb-2                       2/2     Running   0          2d15h
sdk-sample-assembly-dap-idb-0                       2/2     Running   0          2d15h
sdk-sample-assembly-dap-idb-1                       2/2     Running   0          2d15h
sdk-sample-assembly-dap-idb-2                       2/2     Running   0          2d15h
sdk-sample-assembly-dap-rdb-0                       2/2     Running   0          2d15h
sdk-sample-assembly-dap-rdb-1                       2/2     Running   0          2d15h
sdk-sample-assembly-dap-rdb-2                       2/2     Running   0          2d15h
sdk-sample-assembly-sm-0                            5/5     Running   0          2d15h
insights-spctl-sdtransform-v5ma6j9p0o-0             2/2     Running   0          2d15h
insights-spwork-sdtransform-v5ma6j9p0o-1-0          2/2     Running   0          2d15h
rt-sdk-sample-assembly-north-0                      1/1     Running   0          2d15h
rt-sdk-sample-assembly-north-1                      1/1     Running   0          2d15h
rt-sdk-sample-assembly-north-2                      1/1     Running   0          2d15h
rt-sdk-sample-assembly-north-arc-5489dfff9f-88w4h   1/1     Running   0          2d15h
rt-sdk-sample-assembly-south-0                      1/1     Running   0          2d15h
rt-sdk-sample-assembly-south-1                      1/1     Running   0          2d15h
rt-sdk-sample-assembly-south-2                      1/1     Running   0          2d15h
rt-sdk-sample-assembly-south-arc-7d75955d89-8jbq8   1/1     Running   0          2d15h

Client Integration

Now you've deployed the kdb Insights Enterprise along with a sdk-sample-assembly to ingest data, a client must be enrolled to allow data to be published via the authenticated and secure RT entry-point. Here we'll walk through the registration of a simple client while full details on the client enrollment workflow can be found here

The diagram below highlights the key components that allow the flow of data into the system from external sources. To do this securely it is necessary to utilize the Information Service and the Client Controller to register clients and allow them to understand the endpoints. It must also authorize itself and publish the data to the appropriate location

Client Registration

Key to client integration is the ingress endpoint where external clients are accessing the system. The endpoint will have a hostname dependent on the DNS setup of the cluster deploy. In this deployment and sdk-sample-assembly CR the hostname is dev.insights.kx.com and have initialized an ENV INSIGHTS_HOSTNAME to be utilized throughout the working example

export INSIGHTS_HOSTNAME="dev.insights.kx.com"

Client Enrollment

External clients need to be enrolled in the system. This provisions resources for them and configures their topics.

This step requires an access token that can be retrieved as described here.

curl https://${INSIGHTS_HOSTNAME}/clientcontroller/enrol -H "Authorization: Bearer <access token>" -d '{ "name": "client-1", "topics": { "insert": "sdk-sample-assembly", "query": "requests" }}'
{"message":"success","detail":"Client enrolled","url":"d0c8ae0ae920f4ed78b2b8725dffdb74"}

Note

The topics.insert field must match the topic used in the assembly.

In this example, it is sdk-sample-assembly. See the next section for more information.

Matching topics

Topics assigned to clients need to be known by the system. There needs to be an assembly configured to consume that topic otherwise the data will not be published.

To illustrate this the YAML snippet below shows where the topic is configured for an assembly. The sequencer named north is configured as external and its topic is set as sdk-sample-assembly.

spec:
  elements:
    ..
    sequencer:
      ..
      north:
        external: true
        topicConfig:
          subTopic: "sdk-sample-assembly"

Clients configured with the sdk-sample-assembly topic will be matched to this assembly.

Publishing data

The kdb Insights Enterprise SDKs can be used to ingest data into the kdb Insights Enterprise over the Reliable Transport (RT) protocol.

There are two SDKs currently available:

Java SDK.
C SDK.

Removing a client

A client can be removed by making a REST request as below.

Removing a client requires an access token that can be retrieved as described here

curl https://${INSIGHTS_HOSTNAME}/clientcontroller/leave \
     -H "Authorization: Bearer <access token>" \
     -d '{ "name": "client-1" }'
{ "message": "success", "detail": "Client removed" }