Skip to content

Using the sample program for the kdb Insights C interface

This section describes how to use our sample c program, available on Nexus, to:

  • Publish data from the sample.csv file to either:
  • kdb Insights Reliable Transport
  • An sdk_sample_assembly assembly inside the kdb Insights Enterprise. You can download this assembly using the instructions here.
  • Subscribe to data

Downloading

The samples-${VERSION}.zip file, downloaded from the kdb Insights Nexus registry), contains the following files:

  • csvupload.c - csv upload source code
  • CMakeLists.txt - cmake file
  • sample.csv - a CSV file containing sample data based on the sdk-sample-assembly schema

Note

Before using the sample program you must download and extract the C interface.

The source code for the sample program can be taken and edited for use inside your applications.

Building

​ Build the sample programs as follows: ​

unzip kxi-c-sdk-${VERSION}.zip && unzip kxi-c-sdk_samples-${VERSION}.zip
mkdir samples/build && cd samples/build
cmake .. -DKXI_C_SDK_INCLUDE_DIR:PATH=../kxi_c_sdk/include -DKXI_C_SDK_LINK_DIR:PATH=../kxi_c_sdk && cmake --build .

​ This generates an executable csvupload for publishing data and executables test_c, test_k and test_kf for subscribing to data (extensions will vary depending on the platform). ​ ​

Publishing

Application parameters

csvupload demonstrates how to publish data from the sample.csv file and the following parameters are used to configure the application:

parameter required default description
-u str Mandatory none Authenticated client URL
-s str Mandatory none Schema - see note below.
-t str Mandatory none Table (for example: "trace")
-r str Optional /tmp/rt RT log directory
-f int Optional 5000 Time in ms to sleep between reads of the configuration details.
-c int Optional 300000 Maximum age of configuration details in milliseconds. After this amount of time, if still unable to fetch a new configuration, the connection is considered to be broken and any subsequent attempts to send data to kdb Insights Enterprise will fail with a Not connected error. This is only needed when using the kdb Insights Enterprise Information Service to obtain the configuration details.
-l int Optional 600000 Local persistence period (in milliseconds)
-g str Optional info Log Level (example: "info")
-m str Optional err Console log level (info/warn/err/off)
-n str Optional off replicator file log level (trace/debug/info/warn/err/fatal/off)
-w num Optional 2000 Milliseconds to wait for the connection to establish
-i str Optional stdin Used to set an input file name, rather than stdin, eg. '../sample.csv'
-o int Optional 1 Send the file a specific number of times, -1 for infinite loop
-z int Optional 0 Set to 1 to write a single test record
-x str Optional none Absolute or relative path of CA cert file
-d str Optional none DedupId to identify publishers (typically to the same stream) to be deduped
-b int Optional 0 Upload data in batches, sending each batch as a table

The schema defined here MUST match what is defined in the assembly

  • s str: The schema must match that defined in the pipeline assembly. If the schema in the table matches the one in the sdk_sample_assembly, described here, the schema definition is as follows: sensorID:int,captureTS:ts,readTS:ts,valFloat:float,qual:int,alarm:int

Publishing data

The example below runs the csvupload program from the command line. It sets the mandatory parameters and uses defaults for the rest:

url="$KXI_CONFIG_URL"
schema="sensorID:int,captureTS:ts,readTS:ts,valFloat:float,qual:byte,alarm:byte"
table="trace"
./csvupload -u $url -s $schema -t $table < ../sample.csv

url

The url parameter can be set to:

  • KXI_CONFIG_URL to access kdb Insights Enterprise from outside the cluster
  • file:///location/config_file_name to use a local file to access kdb Insights Enterprise or a kdb Insights Reliable Transport from inside the cluster

Subscribing

Application parameters

test_c and test_k demonstrate how to subscribe to data and the following parameters are used to configure the applications:

parameter required default description
str Mandatory none A JSON file name in the URL format that contains RT connection information
long Optional none RT Log position to subscribe from

test_kf demonstrates how to filter subscribed data and the following parameters are used to configure the application:

parameter required default description
str Mandatory none A json file name in the URL format that contains RT connection information
str Mandatory none Table (for example: "trace"), data will be filtered based on this
long Optional none RT Log position to subscribe from

Subscribing to data

The example below runs the subscription test programs from the command line using the following parameters:

  • $KXI_CONFIG_URL - the config URL should be specified as a file URL for a local file that contains the RT connection information.
  • position - the position parameter indicates the position in the RT log from which messages will be subscribed. If absent, this is defaulted to the very first RT message available in the stream logs.
url="$KXI_CONFIG_URL" #(which is in the format file:///<path_to_config>/sub_config.json)
table="trace"
./test_c $url <position>
./test_k $url <position>
./test_kf $url $table <position>

Supported data types

The following data types are supported by the sample application:

type schema notation example
string string name
symbol symbol LLOY
int int 1
byte byte 0
long long 200000
float float 1.2345
timestamp ts 2000.01.01D00:00:00.000000000