Skip to content

Getting started with the kdb Insights SDKs

The kdb Insights SDKs are toolkits created in different languages for developers to integrate with kdb Insights.

You can integrate with kdb Insights Enterprise or the kdb Insights RT Microservice from outside of you cluster via load balancers or from inside you cluster without the need for load balancers.

The SDKs handle the following aspects of the integration:

  • Authentication
  • Connectivity
  • Data transfer
  • Data validation

This section provides details on how to get started using the SDKs.

Using the SDKs

The following SDKs, including sample programs, can be downloaded from the kdb Insights Nexus registry:

Authentication and Connectivity

The SDKs can be used to publish data and query data. They rely on configuration details to determine information about the kdb Insights deployment to connect to, this includes SSL certificates and the endpoints to connect to. For the kdb Insights Enterprise you can provide this configuration via a file or alternatively it can be obtained via another kdb Insights Enterprise service, the Information Service. For kdb Insights RT Microservice you must provide this configuration via a file.

From outside the cluster

Before you can integrate with a kdb Insights Enterprise deployment from outside the cluster, you must be authenticated. If your publisher is outside the kdb Insights Enterprise cluster an Information Service can be called upon to gain the details on the SSL certificates and endpoints. To allow a publisher to use this service, you must follow the authentication steps in this section.

To successfully authenticate, you must have a service account. Create a service account if you do not have one.

Variables

You will need the following variables to generate an authenticated kdb Insights Enterprise client URL (KXI_CONFIG_URL) that allows you to communicate with kdb Insights Enterprise.

variable example further details
INSIGHTS_HOSTNAME kxi-insights.domain.com DNS Hostname setup
REALM_NAME kdbinsights Keycloak realms
KC_CLIENT_ID test-publisher Keycloak Client ID for your service account
KC_CLIENT_SECRET 1R8qtToJNPpt9EuU0qA6MeXZwIXb5RQ5 Keycloak Client secret for your service account

Authentication steps

Follow these steps to generate an authenticated kdb Insights Enterprise client URL (KXI_CONFIG_URL) that allows you to communicate with kdb Insights Enterprise. The KXI_CONFIG_URL provides the SDKs with a TLS key and certificate and the endpoints that they use to connect to the kdb Insights Enterprise and publish or query data.

  1. Obtain an access token for your service account by following the instructions here

  2. Identify the stream id from the External Reference field in the External Data Sources section of the Stream tab of your kdb Insights Database.

    If you are ingesting data directly into the database without any transformations the External Reference will be under the Stream Process section, if you are transforming the data using a pipeline before it is ingested into the database the External Reference will be under the Additional Stream Process section.

    External stream in the UI

    Note

    The screenshot above has both streams set to ingest data from external sources to show the External Reference field in both sections. You might only have one stream that is ingesting data for an external source and therefore only one of these sections will be visible.

  3. Use this access token to retrieve an SDK_CLIENT_UID as follows:

    export SDK_CLIENT_UID=$(curl https://${INSIGHTS_HOSTNAME}/clientcontroller/enrol -H "Authorization: Bearer $INSIGHTS_TOKEN" -d '{ "name": "client-1", "topics": { "<streamid>": "sdk-sample-assembly", "query": "requests" }}' | jq -r '.url')

    {"message":"success","detail":"Client enrolled","url":"${SDK_CLIENT_UID}"}

  4. Create the KXI_CONFIG_URL variable:

    export KXI_CONFIG_URL=https://${INSIGHTS_HOSTNAME}/informationservice/details/${SDK_CLIENT_UID}

From inside the cluster

If you wish to use the SDK in the same cluster as kdb Insights there is no need for an Information Service and a configuration file is used to provide the endpoints and certificates.

The configuration file has the same format as the information passed back from kdb Insights Information Service, but with some important differences:

  • The "useSslRt" top level key needs to be set to false
  • The "ca", "key" and "cert" are no longer needed and will be ignored if provided.
  • You will need to provide the internal hostnames and non-ssl RT port numbers (typically 5002) under the "insert" key

Note

The RT port number (5000 or 5002) refers to the default ports that the internal and external replicators push servers are launched on.

The config file to access an assembly in a kdb Insights Enterprise deployment:

{
    "useSslRt":false, 
    "name":"test",
    "topics":{
            "insert":"<streamid>",
            "query":"requests" },
    "insert": {
            "insert":[":<host1:portnum>",":<host2:portnum>",":<host3:portnum>"],
            "query":[]},
    "query":[":<servicegatewayhost:portnum>"]
}
variable example details
streamid mystream Set in the UI from the External Reference field for the appropriate stream
hostn:portnum :kxi-mystream-2:5000
servicegatewayhost:portnum :kx-insights:6050

The config file to access a kdb Insights RT Microservice deployment:

{
    "useSslRt":false, 
    "name":"test",
    "topics":{
            "insert":"<streamid>",
            "query":"requests" },
    "insert": {
            "insert":[":<host1:portnum>",":<host2:portnum>",":<host3:portnum>"],
            "query":[]},
    "query":[""]
}
variable example details
streamid mystream
hostn:portnum :kxi-mystream-2:5002

Publishing Data

To publish data using an SDK the following are required:

  • A publisher (click on the links below for getting started guides). A publisher can either be:
    • An application that uses one of the following SDKs:
    • An application that sends data to the RT Bridge.
  • A publisher and subscriber that agree on:
    • stream id and parameter usage
    • Message format
  • Sufficient storage: A client publisher can continue running when the RT endpoints are unavailable.

If the endpoints are unavailable at any time

The publisher needs to be aware of the following:

  • Disk space: While the endpoints are unavailable, the data will continue to be written to a local log file on the publisher machine. Therefore there MUST be adequate disk provisioned to handle this build up.

  • Timeout: The SDKs will stop if the endpoints have been offline for an hour.

See details on publishers and subscribers.

Querying Data

Querying data from a kdb Insights Enterprise over IPC is currently only available in the Java SDK.

See here for details on the API calls.