Using the kdb Insights language interfaces
This use case highlights how to deploy the
sdk-sample-assembly Custom Resource (CR) pipeline, publish data into it and query the data, all through secure and authenticated endpoints within kdb Insights Enterprise.
The goal is to take the data pipeline highlighted below and deploy to the kdb Insights Enterprise base application to allow data to be ingested, persisted and queried.
Deploying the Assembly CR
You can deploy the assembly on top of the base instance of the kdb Insights Enterprise in your cluster.
To deploy the assembly, download the
sdk_sample_assembly.yaml from the release repository and deploy to your cluster with the command.
kxi assembly deploy --filepath sdk_sample_assembly.yaml
insights.kx.com/app currently gets set to the name of the assembly so this can be used as a filter to view the resources. For example if you deployed an assembly called
sdk-sample-assembly you can see the pods associated with the assembly by running:
$ kubectl get pods -l insights.kx.com/app=sdk-sample-assembly
NAME READY STATUS RESTARTS AGE
sdk-sample-assembly-dap-hdb-0 2/2 Running 0 2d15h
sdk-sample-assembly-dap-hdb-1 2/2 Running 0 2d15h
sdk-sample-assembly-dap-hdb-2 2/2 Running 0 2d15h
sdk-sample-assembly-dap-idb-0 2/2 Running 0 2d15h
sdk-sample-assembly-dap-idb-1 2/2 Running 0 2d15h
sdk-sample-assembly-dap-idb-2 2/2 Running 0 2d15h
sdk-sample-assembly-dap-rdb-0 2/2 Running 0 2d15h
sdk-sample-assembly-dap-rdb-1 2/2 Running 0 2d15h
sdk-sample-assembly-dap-rdb-2 2/2 Running 0 2d15h
sdk-sample-assembly-sm-0 5/5 Running 0 2d15h
insights-spctl-sdtransform-v5ma6j9p0o-0 2/2 Running 0 2d15h
insights-spwork-sdtransform-v5ma6j9p0o-1-0 2/2 Running 0 2d15h
rt-sdk-sample-assembly-north-0 1/1 Running 0 2d15h
rt-sdk-sample-assembly-north-1 1/1 Running 0 2d15h
rt-sdk-sample-assembly-north-2 1/1 Running 0 2d15h
rt-sdk-sample-assembly-north-arc-5489dfff9f-88w4h 1/1 Running 0 2d15h
rt-sdk-sample-assembly-south-0 1/1 Running 0 2d15h
rt-sdk-sample-assembly-south-1 1/1 Running 0 2d15h
rt-sdk-sample-assembly-south-2 1/1 Running 0 2d15h
rt-sdk-sample-assembly-south-arc-7d75955d89-8jbq8 1/1 Running 0 2d15h
Now you have deployed the kdb Insights Enterprise along with a
sdk-sample-assembly to ingest data, a client must be enrolled to allow data to be published via the authenticated and secure RT entry-point. Here we will walk through the registration of a simple client while full details on the client enrollment workflow can be found here
The diagram below highlights the key components that allow the flow of data into the system from external sources. To do this securely it is necessary to utilize the Information Service and the Client Controller to register clients and allow them to understand the endpoints. It must also authorize itself and publish the data to the appropriate location
Authentication with kdb Insights CLI is a pre-requisite for this step, as described here.
External clients need to be enrolled in the system. This provisions resources for them and configures their topics.
kxi client enrol --name client-1 --insert-topic sdk-sample-assembly
"detail": "Client enrolled",
url field should be stored as it's used by the publisher to get publish endpoints.
$INSIGHTS_HOSTNAME variable is a placeholder for your deployment hostname.
--insert-topic argument must match the topic used in the assembly.
In this example, it is
sdk-sample-assembly. See the next section for more information.
Topics assigned to clients need to be known by the system. There needs to be an assembly configured to consume that topic otherwise the data will not be published.
To illustrate this the YAML snippet below shows where the topic is configured for an assembly. The sequencer named
north is configured as external and its topic is set as
Clients configured with the
sdk-sample-assembly topic will be matched to this assembly.
The kdb Insights interfaces can be used to ingest data. See here for details.
Removing a client
Authentication with cli is a pre-requisite for this step, as described here.
A client can be removed by running the CLI command below:
kxi client remove --name client-1
"detail": "Client removed"