Using the kdb Insights language interfaces
This use case highlights how to deploy the sdk-sample-assembly Custom Resource (CR) pipeline, publish data into it and query the data, all through secure and authenticated endpoints within kdb Insights Enterprise.
The goal is to take the data pipeline highlighted below and deploy to the kdb Insights Enterprise base application to allow data to be ingested, persisted and queried.
Deploying the Assembly CR
You can deploy the assembly on top of the base instance of the kdb Insights Enterprise in your cluster.
To deploy the assembly, download the sdk_sample_assembly.yaml from the release repository and deploy to your cluster with the command.
kxi assembly deploy --filepath sdk_sample_assembly.yaml
The label insights.kx.com/app currently gets set to the name of the assembly so this can be used as a filter to view the resources. For example if you deployed an assembly called sdk-sample-assembly you can see the pods associated with the assembly by running:
$ kubectl get pods -l insights.kx.com/app=sdk-sample-assembly
NAME READY STATUS RESTARTS AGE
sdk-sample-assembly-dap-hdb-0 2/2 Running 0 2d15h
sdk-sample-assembly-dap-hdb-1 2/2 Running 0 2d15h
sdk-sample-assembly-dap-hdb-2 2/2 Running 0 2d15h
sdk-sample-assembly-dap-idb-0 2/2 Running 0 2d15h
sdk-sample-assembly-dap-idb-1 2/2 Running 0 2d15h
sdk-sample-assembly-dap-idb-2 2/2 Running 0 2d15h
sdk-sample-assembly-dap-rdb-0 2/2 Running 0 2d15h
sdk-sample-assembly-dap-rdb-1 2/2 Running 0 2d15h
sdk-sample-assembly-dap-rdb-2 2/2 Running 0 2d15h
sdk-sample-assembly-sm-0 5/5 Running 0 2d15h
insights-spctl-sdtransform-v5ma6j9p0o-0 2/2 Running 0 2d15h
insights-spwork-sdtransform-v5ma6j9p0o-1-0 2/2 Running 0 2d15h
rt-sdk-sample-assembly-north-0 1/1 Running 0 2d15h
rt-sdk-sample-assembly-north-1 1/1 Running 0 2d15h
rt-sdk-sample-assembly-north-2 1/1 Running 0 2d15h
rt-sdk-sample-assembly-north-arc-5489dfff9f-88w4h 1/1 Running 0 2d15h
rt-sdk-sample-assembly-south-0 1/1 Running 0 2d15h
rt-sdk-sample-assembly-south-1 1/1 Running 0 2d15h
rt-sdk-sample-assembly-south-2 1/1 Running 0 2d15h
rt-sdk-sample-assembly-south-arc-7d75955d89-8jbq8 1/1 Running 0 2d15h
Client integration
Now you have deployed the kdb Insights Enterprise along with a sdk-sample-assembly to ingest data.
A client must be enrolled to allow data to be published via the authenticated and secure RT entry-point. Clients can be either users or service accounts. Users and service accounts are created in Keycloak, refer to the Keycloak documentation for more information.
The diagram below highlights the key components that allow the flow of data into the system from external sources. To do this securely it's necessary to utilize the Information Service and the Client Controller to register clients and to allow them to understand the endpoints. It must also authorize itself and publish the data to the appropriate location.
Client enrollment
Prerequisite
A service account or user account must be created in Keycloak and assigned the necessary roles. Your system administrator can set up the user or service account.
Keycloak roles
Here is more information on the relevant Keycloak roles. To enroll a client to publish data to an RT stream, then the insights.client.create and insights.client.delete roles are necessary for the user or service account.
Authenticating as a user
Your system administrator can use the below commands to create a user:
kxi user create demouser --password demoUserPassword --admin-password myAdminPassword
INSIGHTS_ROLES="insights.client.create,insights.client.delete"
kxi user assign-roles demouser --roles $INSIGHTS_ROLES --admin-password myAdminPassword
Now this user can authenticate with the KXI CLI.
kxi auth login # This command launches the browser where demouser can authenticate
Authenticating as service account
Your system administrator can use the below commands to create a service account:
CLIENT_ID=svc1
INSIGHTS_ROLES="insights.client.create,insights.client.delete"
kxi user create-client $CLIENT_ID --admin-password myAdminPassword
kxi user assign-roles service-account-$CLIENT_ID --roles $INSIGHTS_ROLES --admin-password myAdminPassword
CLIENT_SECRET=$(kxi user get-client-secret $CLIENT_ID --admin-password myAdminPassword)
Now admin can provide the above CLIENT_ID and CLIENT_SECRET to any user for configuring their KXI CLI using the below commands.
Service account ID and secret
Your $CLIENT_ID and $CLIENT_SECRET should be inputted to the 'Service account ID' and 'Service account Secret' fields.
kxi configure
Profile type (enterprise, microservices) [enterprise]:
Hostname []: https://{INSIGHTS_HOSTNAME}/
Namespace []: namespace
Service account ID []:
Service account Secret (input hidden):
Re-enter to confirm (input hidden):
Encryption enabled (true/false) [true]:
CLI successfully configured, configuration stored in /home/username/.insights/cli-config
After configuring the kxi with the service account credentials, any user can authenticate as a service account
kxi auth login --serviceaccount
For further details on authentication, refer to the Authentication documentation.
Enroll
- Identify your external reference ID for the RT stream you want to publish to. In this case, it's
sdk-sample-assembly.
External Reference ID
The reference ID can be identified from one of two places:
1. When deploying your RT stream using an assembly or package, ensure that the --insert-topic argument matches the subTopic value from the yaml file.
1. For UI deployments, the reference ID can be found under the Database's Stream Settings tab. The External Reference is found in the External Data Sources section. For the example below, the reference ID is database-demo-e.
spec:
elements:
..
sequencer:
..
north:
external: true
topicConfig:
subTopic: "sdk-sample-assembly"
kxi client enrol --name publisherName --insert-topic sdk-sample-assembly
{
"message": "success",
"detail": "Client enrolled",
"url": "5ed6e5b7c80c8e35d07249d12f32d9eb",
"config_url": "https://{INSIGHTS_HOSTNAME}/informationservice/details/5ed6e5b7c80c8e35d07249d12f32d9eb"
}
The config_url field must be stored as it's used by the publisher to get publish endpoints. The config_url is unique to the publisherName above.
Publishing data
The kdb Insights interfaces can be used to ingest data. Refer to the Interfaces documentation for more details.
Removing a client
Authentication with cli is a pre-requisite for this step, as described here.
A client can be removed by running the CLI command below:
kxi client remove --name publisherName
{
"message": "success",
"detail": "Client removed"
}