Getting started with the interfaces to kdb Insights

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

The kdb Insights interfaces are libraries created in different languages for developers to integrate with kdb Insights Enterprise or the kdb Insights Reliable Transport from outside your cluster via load balancers or from inside your cluster without the need for load balancers.

The interfaces handle the following aspects of the integration:

Authentication
Connectivity
Data transfer
Data validation

You can download the following interfaces, including sample programs, from the KX Downloads Portal or the KX Nexus repository, until Nexus is removed in the future:

This section provides details on how to get started using the interfaces.

Authentication and Connectivity

The interfaces can be used to publish, subscribe, and query data. These interfaces depend on configuration details to establish a connection with the kdb Insights Enterprise deployment. The configuration details include SSL certificates and the endpoints for connection. In the case of kdb Insights Enterprise, you have two options to provide this configuration:

Provide the configuration through a file.
Obtain the configuration from another kdb Insights Enterprise service, specifically the Information Service.

kdb Insights SDK

If you want to use the interfaces within the same cluster as kdb Insights SDK, you can pass the configuration to these interfaces. The interfaces then establish connectivity using their provided helper functions.

The following interfaces include examples of how this configuration can be passed through environment variables or parameters:

kdb Insights Enterprise

Before you can integrate with a kdb Insights Enterprise deployment from outside the cluster, you must be authenticated. If your publisher is outside the kdb Insights Enterprise cluster, the Information Service can be called upon to gain the details on the SSL certificates and endpoints. To allow a publisher to use this service, you must follow the authentication steps in this section.

For details on how you can authenticate to kdb Insights Enterprise as either a service or user account, refer to the Authentication page.

Publishing data

To publish data using an interface the following are required:

A publisher. For getting started guides, refer to the links below. The publisher can be:
An application that uses one of the following interfaces:
- C
- Java
- q (rt.qpk)
- Python
A publisher and subscriber that agree on:
stream id and parameter usage
Message format
Sufficient storage: This allows a client publisher to operate even when the RT endpoints are not available.

If the endpoints are unavailable at any time

The publisher needs to be aware of the following:

Disk space: Data continues to write to a local log file on the publisher machine while the endpoints are unavailable. Therefore, there MUST be adequate disk provisioned to handle this build up.
Timeout: The interface stops if the endpoints have been offline for an hour.

Refer to details on publishers and subscribers.

Querying data

Querying data from either a kdb Insights Enterprise or kdb Insights SDK deployment, over IPC, is currently only available in the Java interface.

Refer to the java SDK documentation for details on the API calls.

Subscribing to data

A subscriber is an application that can receive data. They can use one of the following interfaces: - C - q (rt.qpk)

You must initiate the subscription from within the cluster. The subscriber establishes a connection to the pull_server replicators of RT, which listen on port 5001 by default. For more information about the replicators, refer to the Replicators documentation.

Advanced

If you do not wish to rely on the interface helper functions, you can start the interfaces by passing a configuration file.

The configuration file has the same format as the information passed back from kdb Insights Information Service, but with some important differences:

The "useSslRt" top level key needs to be set to false
The "ca", "key" and "cert" are no longer needed and will be ignored if provided.
You will need to provide the internal hostnames and non-ssl RT port numbers (typically 5002) under the "insert" key

Note

The RT port number (5000, 5001 and 5002) refer to the default ports that the internal and external replicators push servers are launched on.

kdb Insights SDK

Publisher

{
    "useSslRt":false,
    "name":"<client-name>",
    "topics":{
            "insert":"<streamid>",
            "query":"" },
    "insert": {
            "insert":[":<host1:portnum>",":<host2:portnum>",":<host3:portnum>"],
            "query":[]},
    "query":[":<servicegatewayhost:portnum>"]
}

Subscriber

{
    "useSslRt":false,
    "name":"<client-name>",
    "topics":{"subscribe":"<streamid>"},
    "subscribe": {"subscribe":[":<host1:portnum>",":<host2:portnum>",":<host3:portnum>"]}
}

kdb Insights Enterprise

In a kdb Insights Enterprise deployment, you can rely on the Information Service to provide the relevant configuration. Alternatively, the configuration required for kdb Insights Reliable Transport can be provided directly via a json file.

The configuration required for a publisher client is provided below. Note that a kdb Insights Enterprise deployment does not expose the subscription ports by default.

Publisher

{
  "name": "test-client",
  "topics": {
    "insert": "mystream",
    "query": ""
  },
  "ca": "<ca>",
  "cert": "<cert>",
  "key": "<key>",
  "insert": {
    "insert": [
      ":<host1:portnum>",
      ":<host2:portnum>",
      ":<host3:portnum>"
    ],
    "query": []
  },
  "query": [":<servicegatewayhost:portnum>"]
}

Config file tags

Tagnames in the config file are described below:

tagname	variable	example	details
"useSslRt"			Always set to false when publishing to RT from inside the cluster
"name"	client-name	test_client	This name is used to distinguish between two clients publishing from the same host to the same RT stream
"topics":"insert"	streamid	mystream	The RT stream ID that users wish to publish to.
"topics":"subscribe"	streamid	mystream	The RT stream ID that users wish to subscribe to.
"topics":"query"	Not used
"ca"	ca		Certificate of the CA used in the SSL connection
"cert"	cert		Server identity and its public key
"key"	key		Private key for the server's public key
"insert":"insert"	hostn:portnum	:kxi-mystream-2:5000	Array of RT endpoints to publish to, will contain as many endpoints as the RT replicaCount
"subscribe":"subscribe"	hostn:portnum	:kxi-mystream-2:5001	Array of RT endpoints to subscribe to, will contain as many endpoints as the RT replicaCount
"insert":"query"	Not used
"query"	servicegatewayhost:portnum	:kx-insights:6050	Service Gateway endpoint used for querying the database. Not relevant for data ingest through RT.

Note

An [example](../quickstart/kubernetes.md#steps-to-bring-up-rt-chart-with-support-for-external-sdks-and-ssl) is available that demonstrates how to configure SSL end-points for publishing data from outside the cluster.