Skip to content

Using the kdb Insights Java SDK

This section details how to get started with the Java SDK. For examples on how to use it to publish data into kdb Insights Enterprise see documentation on the sample program.

Downloading the Java SDK

You can download the Java SDK from the kdb Insights Maven Nexus registry and a sample program from the kdb Insights package Nexus registry.

Note

The SDKs are only supported on Linux running on x86 architectures. We currently do not support Windows or OsX.

The Java SDK zip file contains everything required to use the Java SDK, you can download the file and unzip.

Environment variables

The Java SDK can be configured through environment variables.

Ingest and Query

variable required description
KXI_CONFIG_URL Mandatory The URL that this program will call to find the endpoint(s) it will need to connect to. This is in the form https://{INSIGHTS_HOSTNAME}/informationservice/details/{KC_CLIENT_ID}.
RT_FILE_LOGLEV Optional Sets the level of logging to file (0-9)
RT_CONSOLE_LOGLEV Optional Sets the level of the console logging (0-9)
RT_DEBUG Optional (deprecated). Setting this to anything other than "0" is the same as setting RT_FILE_LOGLEV=3. Will be ignored is RT_FILE_LOGLEV is set.

Ingest only

variable required description
RT_REP_DIR Mandatory Path where the replicator can be extracted to. It must be possible to execute a program from this location.
RT_LOG_PATH Mandatory The location where the RT message streams will be written to
RT_NO_TRUNCATE Optional If this is set to anything other than "0" log truncation will be disabled

Query only

variable required description
KXI_QUERY_TOKEN Conditional A bearer token used for authentication when making queries. If this is set KXI_QUERY_USER and KXI_QUERY_SECRET are ignored.
KXI_QUERY_USER Conditional The keycloak client that will be used to generate bearer tokens to log in for query Z
KXI_QUERY_SECRET Conditional The keycloak client secret that will be used to generate bearer tokens for query

Much of this configuration can also be performed through code. For example, if you create an instance of StreamingClientFactory with a ConfigRequestor rather than using the default constructor, KXI_CONFIG_URL will be ignored.

Supported data types

The following data types are supported by the Java SDK:

q type name q type number java type schema notation example
boolean -1 Boolean 'boolean' true
guid -2 UUID 'guid' 0f14d0ab-9605-4a62-a9e4-5ed26688389b
byte -4 Byte 'byte' 0xF4
short -5 Short 'short' 5
int -6 Integer 'int' or 'integer' 56789
long -7 Long 'long' 12345678
real -8 Float 'real' 2.5
float -9 Double 'float' 89.3
char -10 Character 'char' d
symbol -11 String 'symbol' LLOY
timestamp -12 java.sql.Timestamp 'timestamp' or 'ts' 2000.01.01D00:00:00.200000000
month -13 kx.c.Month 'month' 2002.02m
date -14 java.sql.Date 'date' 1999.12.31
datetime -15 java.util.Date 'datetime' 12.31,1999.12.31T23:59:59.999
timespan -16 kx.c.Timespan 'timespan' 00:00:00.000000000
minute -17 kc.c.Minute 'minute' 00:00
second -18 kx.c.Second 'second' 00:00:00
time -19 java.sql.Time 'time' 00:00:00
string 10 char[] 'string' "this is a string"

symbol type

Only make text columns into symbols when the fields will be drawn from a small, stable domain and there is significant repetition in their use. When in doubt, start with a string column. It is easier to convert a string column to symbols than it is to remove symbols from the sym list.

time type

Because java.sql.Time does not have milliseconds, some precision is lost if this type is used.

Using the Java SDK in your own application

To publish data to kdb Insights Enterprise using the Java SDK as part of your own application, use the RTClient class. This class reads all the information needed to connect to kdb Insights Enterprise from the environment variables and publishes the data into the system.

Do one of the following:

  • Extract the details from the sample program
  • Follow the sections below that describe how to publish data

Pre-requisites

  • Download the assembly sdk_sample_assembly following the instructions here.
  • Make sure that the assembly is deployed in your kdb Insights Enterprise instance.
  • Ensure you have an authenticated kdb Insights Enterprise client URL.
  • Make sure that the kdb Insights Enterprise ingest endpoints (as defined by the KXI_CONFIG_URL) are accessible.

Initializing

To start a new RtClient object in Java:

RtClient client = new RtClient("$KXI_CONFIG_URL");
client.start();

To pull the configuration URL from the KXI_CONFIG_URL environment variable rather passing it into the constructor, use the StreamingClientFactory:

StreamingClientFactory scf = new StreamingClientFactory(new HttpsConfigRequestor(parser));
StreamingClient client = scf.getStreamingClient();
client.start()

Publishing data

The simplest way to publish data is to use the BulkLoader class. It handles turning 2-dimensional arrays into Flip objects before sending them to kdb Insights Enterprise. You can make one like this (example taken from RandomDataSample.java):

String[] columns = new String[] { "Time", "SensorName", "Voltage", "Current" };
BulkLoader tw = new BulkLoader(tableName, client, columns);

Then you can just pass in 2D arrays, where the first dimension is the row number, and the second is the column number.

Random rand = new Random();
Timestamp now = new Timestamp(System.currentTimeMillis());
for (int i = 0; i < numEntries; i++)
 {
    Object[][] tableToSend = new Object[NUM_SENSORS][];
    for (int j = 0; j < NUM_SENSORS; j++)
    {
        tableToSend[j] = new Object[columns.length];
        tableToSend[j][0] = now;
        tableToSend[j][1] = String.format("Sensor_%d", j);
        tableToSend[j][2] = rand.nextFloat() * 240.0f;
        tableToSend[j][3] = rand.nextFloat() * 10.0f;
        System.out.printf("Writing line, %s, %s, %f, %f\n",
        tableToSend[j][0], tableToSend[j][1], tableToSend[j][2], tableToSend[j][3]);
    }

    tw.writeTable(tableToSend);
}

There are two other loaders available. They both implement the loader interface, so this same basic example applies, but have some differences in how they work:

  • BatchLoader: this will not send data to kdb Insights Enterprise immediately, but will send the data in batches. You can specify the batch size in the constructor.

  • ValidatedLoader: this will check that the data you are sending matches the table schema and throw an exception if it does not match. You will need to obtain an instance of the ValidatedLoader using the ValidatedLoaderFactory. ValidatedLoaderFactory will use the kdb Insights Enterprise getMeta API to read the table schemas for the comparison.

    note!!! The ValidatedLoader assumes that the Stream Processor does not modify the schema of the data before it is written to the table.

Terminating

Once you are done with the client you must call stop() on the RTClient object.

client.stop()