Skip to content

Using the kdb Insights language interfaces

This example demonstrates how to deploy the kxi-db package, publish data to it, and query the data through secure, authenticated endpoints within kdb Insights Enterprise.

The goal is to deploy the package to the kdb Insights Enterprise base application to enable data ingestion, persistence, and querying.

Deploying the package

Deploy the package on top of the base instance of the kdb Insights Enterprise in your cluster.

To deploy the package, download the kxi-db-1.0.0.kxi from the release repository and deploy it to your cluster using the following command:

kxi pm push kxi-db-1.0.0.kxi
kxi pm deploy kxi-db

Client integration

After deployment, kdb Insights Enterprise and kxi-db are ready to ingest data.

A client must be enrolled to allow data to be published through the authenticated and secure RT entry-point. Clients can be either users or service accounts. Users and service accounts are created in Keycloak. For more information, refer to the Keycloak documentation.

The diagram below highlights the key components that allow the flow of data into the system from external sources. To do this securely it's necessary to utilize the Information Service and the Client Controller to register clients and to allow them to understand the endpoints. It must also authorize itself and publish the data to the appropriate location.

Client Registration

Client enrollment

Prerequisite

A service account or user account must be created in Keycloak and assigned the necessary roles. Your system administrator can set up the user or service account.

Keycloak roles

Here is more information on the relevant Keycloak roles. To enroll a client to publish data to an RT stream, then the insights.client.create and insights.client.delete roles are necessary for the user or service account.

Authenticating as a user

Your system administrator can use the below commands to create a user: 

kxi user create demouser --password demoUserPassword --admin-password myAdminPassword
INSIGHTS_ROLES="insights.client.create,insights.client.delete"
kxi user assign-roles demouser --roles $INSIGHTS_ROLES --admin-password myAdminPassword

Now this user can authenticate with the KXI CLI. 

kxi auth login  # This command launches the browser where demouser can authenticate

Authenticating as service account

Your system administrator can use the below commands to create a service account: 

CLIENT_ID=svc1
INSIGHTS_ROLES="insights.client.create,insights.client.delete"
kxi user create-client $CLIENT_ID --admin-password myAdminPassword
kxi user assign-roles service-account-$CLIENT_ID --roles $INSIGHTS_ROLES --admin-password myAdminPassword
CLIENT_SECRET=$(kxi user get-client-secret $CLIENT_ID --admin-password myAdminPassword)

Now admin can provide the above CLIENT_ID and CLIENT_SECRET to any user for configuring their KXI CLI using the below commands.

Service account ID and secret

Your $CLIENT_ID and $CLIENT_SECRET should be inputted to the 'Service account ID' and 'Service account Secret' fields.

kxi configure
Profile type (enterprise, microservices) [enterprise]:
Hostname []: https://{INSIGHTS_HOSTNAME}/
Namespace []: namespace
Service account ID []: 
Service account Secret (input hidden):
Re-enter to confirm (input hidden):
Encryption enabled (true/false) [true]:
CLI successfully configured, configuration stored in /home/username/.insights/cli-config

After configuring the kxi with the service account credentials, any user can authenticate as a service account 

kxi auth login --serviceaccount

For further details on authentication, refer to the Authentication documentation.

Enroll

Identify your external reference ID for the RT stream to publish to. In this example, using the kxi-db package, the external reference ID is ext-mystream.

External Reference ID

The reference ID can be identified from one of two places:

  • When using the kxi pm command to define a package, you can find the reference ID under the subTopic field in the sequencers section of the kxi-db/databases/mydb/shards/mydb-shard.yaml file.

  • For UI deployments, locate the reference ID under the Database's Stream Settings tab. The External Reference is displayed in the External Data Sources section. In the example below, the reference ID is ext-mystream.

External stream in the UI

  1. Enrol the client using the external reference ID.

    kxi client enrol --name publisherName --insert-topic ext-mystream

```shell-session
{
  "message": "success",
  "detail": "Client enrolled",
  "url": "5ed6e5b7c80c8e35d07249d12f32d9eb",
  "config_url": "https://{INSIGHTS_HOSTNAME}/informationservice/details/5ed6e5b7c80c8e35d07249d12f32d9eb"
}

The config_url field must be stored as it's used by the publisher to get publish endpoints. The config_url is unique to the publisherName above.

Publishing data

The kdb Insights interfaces can be used to ingest data. Refer to the Interfaces documentation for more details.

Removing a client

Authentication with cli is a pre-requisite for this step, as described here.

A client can be removed by running the CLI command below:

kxi client remove --name publisherName

{
  "message": "success",
  "detail": "Client removed"
}