Using the kdb Insights Java SDK
This section details how to get started with the Java SDK. For examples on how to use it to publish data into kdb Insights Enterprise see documentation on the sample program.
Downloading the Java SDK
You can download the Java SDK from the kdb Insights Maven Nexus registry and a sample program from the kdb Insights package Nexus registry.
Note
The SDKs are only supported on Linux running on x86 architectures. We currently do not support Windows or OsX.
The Java SDK zip file contains everything required to use the Java SDK, you can download the file and unzip.
Environment variables
The Java SDK can be configured through environment variables.
Ingest and Query
variable | required | description |
---|---|---|
KXI_CONFIG_URL |
Mandatory | The URL that this program will call to find the endpoint(s) it will need to connect to. This is in the form https://{INSIGHTS_HOSTNAME}/informationservice/details/{KC_CLIENT_ID}. |
RT_FILE_LOGLEV |
Optional | Sets the level of logging to file (0-9) |
RT_CONSOLE_LOGLEV |
Optional | Sets the level of the console logging (0-9) |
RT_DEBUG |
Optional | (deprecated). Setting this to anything other than "0" is the same as setting RT_FILE_LOGLEV=3. Will be ignored is RT_FILE_LOGLEV is set. |
Ingest only
variable | required | description |
---|---|---|
RT_REP_DIR |
Mandatory | Path where the replicator can be extracted to. It must be possible to execute a program from this location. |
RT_LOG_PATH |
Mandatory | The location where the RT message streams will be written to |
RT_NO_TRUNCATE |
Optional | If this is set to anything other than "0" log truncation will be disabled |
Query only
variable | required | description |
---|---|---|
KXI_QUERY_TOKEN |
Conditional | A bearer token used for authentication when making queries. If this is set KXI_QUERY_USER and KXI_QUERY_SECRET are ignored. |
KXI_QUERY_USER |
Conditional | The keycloak client that will be used to generate bearer tokens to log in for query Z |
KXI_QUERY_SECRET |
Conditional | The keycloak client secret that will be used to generate bearer tokens for query |
Much of this configuration can also be performed through code. For example, if you create an instance of StreamingClientFactory with a ConfigRequestor rather than using the default constructor, KXI_CONFIG_URL
will be ignored.
Supported data types
The following data types are supported by the Java SDK:
q type name | q type number | java type | schema notation | example |
---|---|---|---|---|
boolean | -1 | Boolean | 'boolean' | true |
guid | -2 | UUID | 'guid' | 0f14d0ab-9605-4a62-a9e4-5ed26688389b |
byte | -4 | Byte | 'byte' | 0xF4 |
short | -5 | Short | 'short' | 5 |
int | -6 | Integer | 'int' or 'integer' | 56789 |
long | -7 | Long | 'long' | 12345678 |
real | -8 | Float | 'real' | 2.5 |
float | -9 | Double | 'float' | 89.3 |
char | -10 | Character | 'char' | d |
symbol | -11 | String | 'symbol' | LLOY |
timestamp | -12 | java.sql.Timestamp | 'timestamp' or 'ts' | 2000.01.01D00:00:00.200000000 |
month | -13 | kx.c.Month | 'month' | 2002.02m |
date | -14 | java.sql.Date | 'date' | 1999.12.31 |
datetime | -15 | java.util.Date | 'datetime' | 12.31,1999.12.31T23:59:59.999 |
timespan | -16 | kx.c.Timespan | 'timespan' | 00:00:00.000000000 |
minute | -17 | kc.c.Minute | 'minute' | 00:00 |
second | -18 | kx.c.Second | 'second' | 00:00:00 |
time | -19 | java.sql.Time | 'time' | 00:00:00 |
string | 10 | char[] | 'string' | "this is a string" |
symbol type
Only make text columns into symbols when the fields will be drawn from a small, stable domain and there is significant repetition in their use. When in doubt, start with a string column. It is easier to convert a string column to symbols than it is to remove symbols from the sym list.
time type
Because java.sql.Time does not have milliseconds, some precision is lost if this type is used.
Using the Java SDK in your own application
To publish data to kdb Insights Enterprise using the Java SDK as part of your own application, use the RTClient
class. This class reads all the information needed to connect to kdb Insights Enterprise from the environment variables and publishes the data into the system.
Do one of the following:
- Extract the details from the sample program
- Follow the sections below that describe how to publish data
Pre-requisites
- Download the assembly
sdk_sample_assembly
following the instructions here. - Make sure that the assembly is deployed in your kdb Insights Enterprise instance.
- Ensure you have an authenticated kdb Insights Enterprise client URL.
- Make sure that the kdb Insights Enterprise ingest endpoints (as defined by the
KXI_CONFIG_URL
) are accessible.
Initializing
To start a new RtClient object in Java:
RtClient client = new RtClient("$KXI_CONFIG_URL");
client.start();
To pull the configuration URL from the KXI_CONFIG_URL environment variable rather passing it into the constructor, use the StreamingClientFactory:
StreamingClientFactory scf = new StreamingClientFactory(new HttpsConfigRequestor(parser));
StreamingClient client = scf.getStreamingClient();
client.start()
Publishing data
The simplest way to publish data is to use the BulkLoader class. It handles turning 2-dimensional arrays into Flip objects before sending them to kdb Insights Enterprise. You can make one like this (example taken from RandomDataSample.java):
String[] columns = new String[] { "Time", "SensorName", "Voltage", "Current" };
BulkLoader tw = new BulkLoader(tableName, client, columns);
Then you can just pass in 2D arrays, where the first dimension is the row number, and the second is the column number.
Random rand = new Random();
Timestamp now = new Timestamp(System.currentTimeMillis());
for (int i = 0; i < numEntries; i++)
{
Object[][] tableToSend = new Object[NUM_SENSORS][];
for (int j = 0; j < NUM_SENSORS; j++)
{
tableToSend[j] = new Object[columns.length];
tableToSend[j][0] = now;
tableToSend[j][1] = String.format("Sensor_%d", j);
tableToSend[j][2] = rand.nextFloat() * 240.0f;
tableToSend[j][3] = rand.nextFloat() * 10.0f;
System.out.printf("Writing line, %s, %s, %f, %f\n",
tableToSend[j][0], tableToSend[j][1], tableToSend[j][2], tableToSend[j][3]);
}
tw.writeTable(tableToSend);
}
There are two other loaders available. They both implement the loader interface, so this same basic example applies, but have some differences in how they work:
-
BatchLoader: this will not send data to kdb Insights Enterprise immediately, but will send the data in batches. You can specify the batch size in the constructor.
-
ValidatedLoader: this will check that the data you are sending matches the table schema and throw an exception if it does not match. You will need to obtain an instance of the ValidatedLoader using the ValidatedLoaderFactory. ValidatedLoaderFactory will use the kdb Insights Enterprise getMeta API to read the table schemas for the comparison.
note!!! The ValidatedLoader assumes that the Stream Processor does not modify the schema of the data before it is written to the table.
Terminating
Once you are done with the client you must call stop()
on the RTClient object.
client.stop()