Skip to content

Getting Starting with the kdb Insights SDKs

The kdb Insights SDKs are toolkits created in different languages for developers to integrate with kdb Insights Enterprise from outside of the cluster. They handle the following aspects of the integration:

  • Authentication
  • Connectivity
  • Data transfer
  • Data validation

This section provides details on how to get started using the SDKs.

Using the SDKs

The following SDKs, can be embedded in your applications to allow you to publish data into, and query data out of kdb Insights Enterprise securely.

Prerequisites

You must have a service account to successfully authenticate with the SDKs, create a service account if you do not have one.

Authentication

Before you can ingest or query a kdb Insights Enterprise deployment, you must be authenticated.

Variables

You will need the following variables to generate an authenticated kdb Insights Enterprise client URL (KXI_CONFIG_URL) that allows you to communicate with kdb Insights Enterprise.

variable example further details
INSIGHTS_HOSTNAME kxi-insights.domain.com DNS Hostname setup
REALM_NAME kdbinsights Keycloak realms
KC_CLIENT_ID test-publisher Keycloak Client ID for your service account
KC_CLIENT_SECRET 1R8qtToJNPpt9EuU0qA6MeXZwIXb5RQ5 Keycloak Client secret for your service account

Follow these steps to generate an authenticated kdb Insights Enterprise client URL (KXI_CONFIG_URL) that allows you to communicate with kdb Insights Enterprise. The KXI_CONFIG_URL provides the SDKs with a TLS key and certificate and the endpoints that they use to connect to the kdb Insights Enterprise and publish or query data.

  1. Obtain an access token for your service account:

    export INSIGHTS_TOKEN=$(curl --header "Content-Type: application/x-www-form-urlencoded" -d "grant_type=client_credentials&client_id=$KC_CLIENT_ID&client_secret=$KC_CLIENT_SECRET" "https://${INSIGHTS_HOSTNAME}/auth/realms/${REALM_NAME}/protocol/openid-connect/token" | jq -r '.access_token')
    
  2. Identify the [stream id] from the Sub Topic defined for the externally facing RT (or KX Insights Stream) deployed in the assembly.

  3. Use this access token to retrieve an SDK_CLIENT_UID as follows:

    export SDK_CLIENT_UID=$(curl https://${INSIGHTS_HOSTNAME}/clientcontroller/enrol -H "Authorization: Bearer $INSIGHTS_TOKEN" -d '{ "name": "client-1", "topics": { "<streamid>": "sdk-sample-assembly", "query": "requests" }}' | jq -r '.url')
    
    {"message":"success","detail":"Client enrolled","url":"${SDK_CLIENT_UID}"}
    
  4. Create the KXI_CONFIG_URL variable:

    export KXI_CONFIG_URL=https://${INSIGHTS_HOSTNAME}/informationservice/details/${SDK_CLIENT_UID}
    

Publishing Data

To publish data into kdb Insights Enterprise using an SDK the following are required:

  • A publisher (click on the links below for getting started guides). A publisher can either be:
    • An application that uses one of the following SDKs:
    • An application that sends data to the RT Bridge to publish into kdb Insights Enterprise.
  • A publisher and subscriber that agree on:
    • stream id and parameter usage
    • Message format
  • Sufficient Storage: A client publisher can continue running when an assembly is torn down or kdb Insights Enterprise is offline. The SDKs regularly poll the Information Service for details on the SSL certs and the RT endpoints. If the Information Service or the endpoints are unavailable the kdb Insights SDKs continue to write to the publisher's local disk and they ship the queued data as and when the assembly becomes available. 

If the KX Insights Stream is unavailable at any time

The publisher needs to be aware of the following:

  • Disk space: While the KX Insights Stream is unavailable, the data will continue to be written to a local log file on the publisher machine. Therefore there MUST be adequate disk provisioned to handle this build up.

  • Timeout: The SDKs will stop if the stream has been offline for an hour.

Querying Data

Querying data from a kdb Insights Enterprise over IPC is currently only available in the Java SDK.

See here for details on the API calls.