Skip to content

Service Gateway APIs

These pre-configured APIs are available via q IPC or REST

.kxi. getMeta meta information on APIs defined in the DAPs and aggregation functions getData generic data retrieval qsql execute qSQL query ping diagnostic tool to determine system health

DAP implementation

For each API, the Data Access processes (DAPs) must implement the functions specified in the description of the API.

If using the Service Gateway (SG) with the KX Insights Data Access microservice, these APIs are already implemented and no further work is required to use them.

In the examples we assume the Service Gateway (SG) assembly is

name: mySG
labels:
  region: amer,emea,apac
elements:
  rc:
    host: rcHost
    port: 1234

and the Data Access processes (DAPs) the SG fronts contain a table called trace with schema

sensorID   int         sensor identifier
readTS     timestamp   timestamp of reading HDB partition column
valFloat   float       reading measurment
qual       short       quality of the reading
updateTS   timestamp   update timestamp of reading

Finally, we assume

  • the REST server’s address is stored in an environment variable URL (e.g. URL=localhost:5030) when making REST calls
  • we have an open handle h to a GW process when making q IPC requests

Purview parameters

All API calls may include startTS, endTS and all assembly labels in the parameters for the SG to route to the correct DAPs.

In these examples, with the assembly specified above, we therefore may include region in all API calls. We may choose to target a single region, or multiple, e.g.

params:`startTS`endTS`region!(-0Wp;0Wp;`amer) / Single region
params:`startTS`endTS`region!(-0Wp;0Wp;`amer`emea) / Multiple regions

If a label, startTS, or endTS is ommitted from the parameters, it defaults to "all". For example:

params:enlist[`startTS]!enlist 2021.08.05D

is equivalent to

params:`startTS`endTS`region!(2021.08.05D;0Wp;`amer`emea`apac)

.kxi.getData

A generic data-retrieval API that can be executed against an arbitrary database table

.kxi.getData args

Where args is a dictionary:

table    name of table to retrieve data from (symbol)
startTS  inclusive start time of period of interest (timestamp)
endTS    inclusive end time of period of interest (timestamp)
filter   qSQL where clause defining filter (string)
columns  Columns to return in response (symbol vector)
opts     further options (dictionary)

and opts is a dictionary:

tsCol      timestamp column to apply startTS/endTS to (symbol);
           default is the partition column of table
sortCols   columns to sort response payload on (symbol vector)

returns a 2-item list

  1. response header
  2. table of data

The DAP must implement .kxi.getData. The response is a 3-item list containing

  1. columns to sort on or (::) for none (symbol vector)
  2. columns to select (symbol vector)
  3. table data, with at least the union of columns in the first two elements
http://${URL}/data

A basic Get call with only the table and time range specified:

q)args
table  | `trace
startTS| -0Wp
endTS  | 0Wp
region | amer

Sync:

q)res:h(`.kxi.getData;args;`;(0#`)!()) / Make API call and store the result in "res"
q)5#last res / Display only the first 5 records
sensorID readTS                        valFloat qual updateTS
----------------------------------------------------------------------------------
0        2021.02.09D00:00:00.000000000 283.0992 1    2021.02.10D19:02:25.332557500
0        2021.02.09D01:00:00.000000000 906.6764 5    2021.02.10D19:02:25.332557500
0        2021.02.09D02:00:00.000000000 546.6978 4    2021.02.10D19:02:25.332557500
0        2021.02.09D03:00:00.000000000 164.0931 4    2021.02.10D19:02:25.332557500
0        2021.02.09D04:00:00.000000000 663.1649 4    2021.02.10D19:02:25.332557500

Async:

q)f:{res::y} / Store only the payload in "res"
q)neg[h](`.kxi.getData;args;`f;(0#`)!()) / Make API call and invoke callback
q)5#res / Display only the first 5 records
sensorID readTS                        valFloat qual updateTS
----------------------------------------------------------------------------------
0        2021.02.09D00:00:00.000000000 283.0992 1    2021.02.10D19:02:25.332557500
0        2021.02.09D01:00:00.000000000 906.6764 5    2021.02.10D19:02:25.332557500
0        2021.02.09D02:00:00.000000000 546.6978 4    2021.02.10D19:02:25.332557500
0        2021.02.09D03:00:00.000000000 164.0931 4    2021.02.10D19:02:25.332557500
0        2021.02.09D04:00:00.000000000 663.1649 4    2021.02.10D19:02:25.332557500
curl -X POST\
    --header "Content-Type: application/json"\
    --data '{"table":"trace", "startTS":"2000.01.01D", "endTS":"2099.01.01D", "region": "amer"}'
    "http://${URL}/data"

Refine the time range by specifying startTS:

q)args
table  | `trace
startTS| 2021.02.09D01:00:00.000000000
endTS  | 0Wp
region | `emea

Sync:

q)/ Make the API call and store the result in "res"
q)res:h(`.kxi.getData;args;`;(0#`)!())
q)5#last res / Display only the first 5 records
sensorID readTS                        valFloat qual updateTS
----------------------------------------------------------------------------------
0        2021.02.09D01:00:00.000000000 906.6764 5    2021.02.10D19:02:25.332557500
0        2021.02.09D02:00:00.000000000 546.6978 4    2021.02.10D19:02:25.332557500
0        2021.02.09D03:00:00.000000000 164.0931 4    2021.02.10D19:02:25.332557500
0        2021.02.09D04:00:00.000000000 663.1649 4    2021.02.10D19:02:25.332557500
0        2021.02.09D05:00:00.000000000 537.0982 4    2021.02.10D19:02:25.332557500

Async:

q)f:{res::y} / Store only payload in res
q)neg[h](`.kxi.getData;args;`f;(0#`)!()) / Make the API call and invoke callback
q)5#last res / Display only the first 5 records
sensorID readTS                        valFloat qual updateTS
----------------------------------------------------------------------------------
0        2021.02.09D01:00:00.000000000 906.6764 5    2021.02.10D19:02:25.332557500
0        2021.02.09D02:00:00.000000000 546.6978 4    2021.02.10D19:02:25.332557500
0        2021.02.09D03:00:00.000000000 164.0931 4    2021.02.10D19:02:25.332557500
0        2021.02.09D04:00:00.000000000 663.1649 4    2021.02.10D19:02:25.332557500
0        2021.02.09D05:00:00.000000000 537.0982 4    2021.02.10D19:02:25.332557500
curl -X POST\
    --header "Content-Type: application/json"\
    --data '{"table":"trace", "startTS":"2021.02.09D01", "endTS":"2099.01.01D", "region": "emea"}'
    "http://${URL}/data"

Select only a subset of the columns and multiple regions:

q)args
table  | `trace
startTS| -0Wp
endTS  | 0Wp
region | `amer`emea
columns| `sensorID`readTS`valFloat

Sync:

q)/ Make the API call and store the result in "res"
q)res:h(`.kxi.data;args;`;(0#`)!())
q)5#last res / Display only the first 5 records
sensorID readTS                        valFloat
-----------------------------------------------
0        2021.05.12D00:00:00.000000000 278.1034
0        2021.05.12D01:00:00.000000000 256.3526
0        2021.05.12D02:00:00.000000000 73.38756
0        2021.05.12D03:00:00.000000000 911.9714
0        2021.05.12D04:00:00.000000000 171.8527

Async:

q)f:{res::y} / Store only the payload in "res"
q)res:neg[h](`.kxi.data;args;`f;(0#`)!()) / Make the API call and invoke callback
q)5#res / Display only the first 5 records
sensorID readTS                        valFloat
-----------------------------------------------
0        2021.05.12D00:00:00.000000000 278.1034
0        2021.05.12D01:00:00.000000000 256.3526
0        2021.05.12D02:00:00.000000000 73.38756
0        2021.05.12D03:00:00.000000000 911.9714
0        2021.05.12D04:00:00.000000000 171.8527
curl -X POST\
    --header "Content-Type: application/json"\
    --data '{"table":"trace", "startTS":"2000.01.01D", "endTS":"2099.01.01D", "region": ["amer", "emea"], "columns": ["sensorID", "readTS", "valFloat"]}'
    "http://${URL}/data"

Use the filter argument to ask for sensor IDs equal to 10i and qual values between 1 and 3h:

q)args
table  | `trace
startTS| -0Wp
endTS  | 0Wp
region | apac
filter | "sensorID=10i,qual within 1 3h"

Sync:

q)/ Make the API call and store the result in "res"
q)res:h(`.kxi.getData;args;`;(0#`)!())
q)5#last res / Display only the first 5 records
sensorID readTS                        valFloat qual updateTS
----------------------------------------------------------------------------------
10       2021.05.12D03:00:00.000000000 781.8174 3    2021.05.13D14:01:56.802276100
10       2021.05.12D04:00:00.000000000 951.9392 2    2021.05.13D14:01:56.802276100
10       2021.05.12D07:00:00.000000000 971.0995 3    2021.05.13D14:01:56.802276100
10       2021.05.12D13:00:00.000000000 103.5567 2    2021.05.13D14:01:56.802276100
10       2021.05.12D14:00:00.000000000 297.4721 1    2021.05.13D14:01:56.802276100

Async:

q)f:{res::y} / Store only the payload in "res"
q)/ Make the API call and store the result in "res"
q)res:h(`.kxi.getData;args;`f;(0#`)!())
q)5#last res / Display only the first 5 records
sensorID readTS                        valFloat qual updateTS
----------------------------------------------------------------------------------
10       2021.05.12D03:00:00.000000000 781.8174 3    2021.05.13D14:01:56.802276100
10       2021.05.12D04:00:00.000000000 951.9392 2    2021.05.13D14:01:56.802276100
10       2021.05.12D07:00:00.000000000 971.0995 3    2021.05.13D14:01:56.802276100
10       2021.05.12D13:00:00.000000000 103.5567 2    2021.05.13D14:01:56.802276100
10       2021.05.12D14:00:00.000000000 297.4721 1    2021.05.13D14:01:56.802276100
curl -X POST\
    --header "Content-Type: application/json"\
    --data '{"table":"trace", "startTS":"2000.01.01D", "endTS":"2099.01.01D", "region": "apac", "filter": "sensorID=10i,qual within 1 3h"}'
    "http://${URL}/data"

Use the options dictionary to apply the timestamp filter to the updateTS column instead:

q)args
table  | `trace
startTS| 2021.05.13D14:05:00.000000000
region | amer
endTS  | 0Wp
opts   | (,`tsCol)!,`updateTS

Sync:

q)/ Make the API call and store the result in "res"
q)res:h(`.kxi.getData;args;`;(0#`)!())
q)5#last res / Display only the first 5 records
sensorID readTS                        valFloat qual updateTS
----------------------------------------------------------------------------------
0        2021.05.12D00:00:00.000000000 458.3568 5    2021.05.13D14:05:06.802253100
0        2021.05.12D01:00:00.000000000 919.25   3    2021.05.13D14:05:06.802253100
0        2021.05.12D02:00:00.000000000 465.7557 1    2021.05.13D14:05:06.802253100
0        2021.05.12D03:00:00.000000000 609.1758 2    2021.05.13D14:05:06.802253100
0        2021.05.12D04:00:00.000000000 349.6804 1    2021.05.13D14:05:06.802253100

Async:

q)f:{res::y} / Store only the payload in "res"
q)neg[h](`.kxi.getData;args;`f;(0#`)!()) / Make the API call and invoke callback
q)5#res / Display only the first 5 records
sensorID readTS                        valFloat qual updateTS
----------------------------------------------------------------------------------
0        2021.05.12D00:00:00.000000000 458.3568 5    2021.05.13D14:05:06.802253100
0        2021.05.12D01:00:00.000000000 919.25   3    2021.05.13D14:05:06.802253100
0        2021.05.12D02:00:00.000000000 465.7557 1    2021.05.13D14:05:06.802253100
0        2021.05.12D03:00:00.000000000 609.1758 2    2021.05.13D14:05:06.802253100
0        2021.05.12D04:00:00.000000000 349.6804 1    2021.05.13D14:05:06.802253100
curl -X POST\
    --header "Content-Type: application/json"\
    --data '{"table":"trace", "startTS":"2021.05.13D14:05", "endTS":"2099.01.01D", "region": "amer", "opts": {"tsCol": "updateTS"}}'
    "http://${URL}/data"

Use the options dictionary to have the result sorted by qual and valFloat:

q)args
table  | `trace
startTS| 2021.05.13D14:05:00.000000000
endTS  | 0Wp
region | `amer
opts   | (,`sortCols)!,`qual`valFloat

Sync:

q)/ Make the API call and store the result in "res"
q)res:h(`.kxi.getData;args;`;(0#`)!())
q)5#last res / Display only the first 5 records
sensorID readTS                        valFloat   qual updateTS
------------------------------------------------------------------------------------
99       2021.05.12D08:00:00.000000000 0.00825664 0    2021.05.13D14:11:56.802232800
39       2021.05.12D12:00:00.000000000 0.09356276 0    2021.05.13D14:12:36.802253700
70       2021.05.12D02:00:00.000000000 0.1157809  0    2021.05.13D14:17:56.802235900
1        2021.05.12D01:00:00.000000000 0.197002   0    2021.05.13D14:17:46.804990700
33       2021.05.12D12:00:00.000000000 0.2034092  0    2021.05.13D14:13:16.803428200

Async:

q)f:{res::y} / Store only the payload in "res"
q)neg[h](`.kxi.getData;args;`f;(0#`)!()) / Make the API call and invoke callback
q)5#res / Display only the first 5 records
sensorID readTS                        valFloat   qual updateTS
------------------------------------------------------------------------------------
99       2021.05.12D08:00:00.000000000 0.00825664 0    2021.05.13D14:11:56.802232800
39       2021.05.12D12:00:00.000000000 0.09356276 0    2021.05.13D14:12:36.802253700
70       2021.05.12D02:00:00.000000000 0.1157809  0    2021.05.13D14:17:56.802235900
1        2021.05.12D01:00:00.000000000 0.197002   0    2021.05.13D14:17:46.804990700
33       2021.05.12D12:00:00.000000000 0.2034092  0    2021.05.13D14:13:16.803428200
curl -X POST\
    --header "Content-Type: application/json"\
    --data '{"table":"trace", "startTS":"2021.05.13D14:05", "endTS":"2099.01.01D", "region": "amer", "opts": {"sortCols": ["qual", "valFloat"]}}'
    "http://${URL}/data"

.kxi.getMeta

Meta information on APIs defined in the DAPs and aggregation functions

.kxi.getMeta args

Where args is a dictionary containing any purview parameters, returns a two-item list:

  1. response header
  2. dictionary with keys:
api        table of APIs available in DAPs
agg        table of Agg functions available in Aggs
assembly   table of assemblies fronted by the SG
schema     table of schemas in assemblies fronted by the SG

The table schemas are

api

The set of APIs and what labels/SGs offer them

column type description
api symbol Name of API.
label_1 symbol[] Which values of label_1 support this API.
label_n symbol[] Which values of label_n support this API.
aggFn symbol Default aggregation function (where null means raze).
custom boolean Whether this is a custom API (true) or a built-in KXI API (false).
full boolean True if all DAPs/Aggs for these labels support this API and agg function mapping.
metadata dictionary JSON description of the API (see "SAPI - Metadata").
procs table Table of individual DAP and Agg processes corresponding to this entry if not full.

If full is true, the procs cell is empty, otherwise a table of all the DAPs and Aggs to which this API and agg function mapping apply. You can use it to define preferential routing to access specific versions of the API, and/or determine which labels to target in API arguments. The schema of the table is:

dap         symbol    host/port of DAP
agg         symbol    host/port of Agg
sg          symbol    sg of DAP/Agg
label_1     symbol    value of label_1 of DAP
...         ...       ...
label_n     symbol    value of label_n of DAP

Besides the usual metadata keys (description, params, return), an API’s metadata contains one further key: aggReturn. While return refers to the return of the actual function from the DAP (what Agg receives), aggReturn is the return of the corresponding default aggregation function (what the caller receives back from the GW).

Note that we may have more than one row per API if different labels/SGs/Aggs support different versions of the API. E.g.

api   region     decription                     ..
------------------------------------------------..
myAPI ,`amer     "amer version of this API"     ..
myAPI `emea`apac "emea/apac version of this API"..
..

agg

The set of aggregation functions and what labels/SGs offer them

column type description
aggFn symbol Name of aggregation function.
label_1 symbol[] Which values of label_1 support this aggregation function.
... ... ...
label_n symbol[] Which values of label_n support this aggregation function.
custom boolean Whether this is a custom aggrgation function (true) or built-in (false).
full boolean True if all DAPs/Aggs of these labels/SGs support this API/defaultAgg mapping.
metadata dictionary JSON description of the aggregation function (see "SAPI - Metadata").
procs table Table of individual DAP and Agg processes corresponding to this entry if not full.

If full is true, the procs cell is empty, otherwise a table of all the Aggs that contain this aggregation function. This can be used to define preferential routing to access specific versions of the API, and/or determine which labels to target in API arguments. The schema of the table is:

agg        symbol    host/port of Agg
sg         symbol    sg of Agg
label_1    symbol[]  values of label_1 of Agg
...        ...       ...
label_n    symbol[]  values of label_n of Agg

Example:

q)args:`region`startTS`endTS!(`amer`emea`apac;-0Wp;0Wp)
q)res:h(`.kxi.getMeta;args;`;(0#`)!())
q)res`api
api region         aggFn  custom full metadata                                                                                      ..
------------------------------------------------------------------------------------------------------------------------------------..
fn1 amer apac emea aggFn1 0      1    `description`params`return`aggReturn!("Function 1";+(,`name)!,`x`y;`type`description!(98h;"res..
fn2 amer apac emea aggFn1 0      1    `description`params`return`aggReturn!("Function 2";(`name`type!(`arg1;11h);`name`isReq!(`arg2;..
fn3 amer apac emea        1      1    `description`params`return`aggReturn!("";();(`symbol$())!();(`symbol$())!())                  ..

q)res`aggFn
aggFn  region         custom full metadata                                                                                          ..
------------------------------------------------------------------------------------------------------------------------------------..
aggFn1 amer apac emea 0      1    `description`params`return!("";+`name`type!(,`list;,0h);(`symbol$())!())                          ..
aggFn2 amer apac emea 1      1    `description`params`return!("Agg function 2";+(,`type)!,,6 7 8 9h;`type`description!(-9h;"Sum"))  ..

assembly

The assemblies fronted by the SG

column type description
assembly symbol Assembly name.
label_1 symbol Value of label_1 for this assembly.
... ... ...
label_n symbol Value of label_n for this assembly.

schema

The tables contained in the assemblies fronted by the SG

column type description
table symbol Table name.
assembly symbol[] Assemblies that contain this table with this schema.
typ symbol Table type.
pkCols symbol[] Primary key columns.
updTsCol symbol Update time column.
prtnCol symbol Column the data is partitioned on.
sortColsMem symbol[] Columns the data is sorted on in memory (e.g. RDB).
sortColsIDisk symbol[] Columns the data is sorted on on-intraday-disk (e.g. IDB).
sortColsDisk symbol[] Columns the data is sorted on on-disk (e.g. HDB).
columns table Table columns (see below).

Each columns cell is a table with the following columns:

column type description example
column symbol Column name. `price
typ short KDB+ type. 9h
attrMem symbol In-memory attribute (e.g. RDB). `
attrIDisk symbol On-intraday-disk attribute (e.g. IDB). `s
attrDisk symbol On-disk attribute (e.g. HDB). `p
isSerialized boolean Whether column is serialized. 0b
fk symbol Foreign table information. instrument.sym

A table may appear in multiple rows if it appears in multiple assemblies with a different schema/property:

table assembly typ        ..
--------------------------..
trade ,`asm1   partitioned..
trade ,`asm2   splayed    ..
..

Example:

q)args:`region`startTS`endTS!(`amer`emea`apac;-0Wp;0Wp)
q)res:h(`.kxi.getMeta;args;`;(0#`)!())
q)res`api
api region         aggFn  custom full metadata                                                                                       ..
-------------------------------------------------------------------------------------------------------------------------------------..
fn1 amer apac emea aggFn1 0      1    `description`params`return`aggReturn!("Function 1";+(,`name)!,`x`y;`type`description!(98h;"res"..
fn2 amer apac emea aggFn1 0      1    `description`params`return`aggReturn!("Function 2";(`name`type!(`arg1;11h);`name`isReq!(`arg2;0..
fn3 amer apac emea        1      1    `description`params`return`aggReturn!("";();(`symbol$())!();(`symbol$())!())                   ..

q)res`aggFn
aggFn  region         custom full metadata                                                                                          ..
------------------------------------------------------------------------------------------------------------------------------------..
aggFn1 amer apac emea 0      1    `description`params`return!("";+`name`type!(,`list;,0h);(`symbol$())!())                          ..
aggFn2 amer apac emea 1      1    `description`params`return!("Agg function 2";+(,`type)!,,6 7 8 9h;`type`description!(-9h;"Sum"))  ..

q)res`assembly // Assume one assembly per region
assembly region
---------------
amerAsm  amer
emeaAsm  emea
apacAsm  apac

q)res`schema // Basic tick schema
table typ         pkCols updTsCol columns                                                                                                         ..
--------------------------------------------------------------------------------------------------------------------------------------------------..
trade partitioned                 (+(,`column)!,`time`sym`realTime`price`size)!+`typ`attrMem`attrIDisk`attrDisk`isSerialized`fk!(-16 -11 -12 -9 -7..
quote partitioned                 (+(,`column)!,`time`sym`realTime`bid`ask`bidSize`askSize)!+`typ`attrMem`attrIDisk`attrDisk`isSerialized`fk!(-16 ..

.kxi.ping

A diagnostic tool to determine system health

.kxi.ping args

Where args is a dictionary (can be empty), returns a boolean vector of where the DAP reached its target.

The DAP must implement .kxi.ping. The function should return 1b.

http://${URL}/ping

In a system with an RDB/IDB/HDB, a call to one region for all time should hit all three.

q)args
startTS| -0Wp
endTS  | 0Wp
region | `amer

Sync:

q)/ Make the API call and store the result in "res"
q)res:h(`.kxi.ping;args;`;(0#`)!())
q)5#res

Async:

q)f:{res::y} / Store only the payload in "res"
q)neg[h](`.kxi.getData;args;`f;(0#`)!()) / Make the API call and invoke callback
q)res
111b
curl -X POST\
    --header "Content-Type: application/json"\
    --data '{"startTS":"2000.01.01D", "endTS":"2099.01.01D", "region": "amer"}'
    "http://${URL}/ping"

.kxi.qsql

.kxi.qsql args

Where args is a dictionary

query   qSQL query (string)

executes the query on the DAP and returns a 2-item list

  1. response header
  2. query result, typically a table

The DAP must implement .kxi.qsql. The function should return the result of executing the query string.

http://${URL}/ping

A simple select from the trace table for the sensor with ID 4:

q)args
startTS| -0Wp
endTS  | 0Wp
region | `amer
query  | "select from trace where sensorID=4"

Sync:

q)res:h(`.kxi.qsql;args;`;(0#`)!()) / Make the API call and
q)5#last res / Display only the first 5 records
sensorID readTS                        valFloat qual updateTS
----------------------------------------------------------------------------------
4        2021.02.09D00:00:00.000000000 283.0992 1    2021.02.10D19:02:25.332557500
4        2021.02.09D01:00:00.000000000 906.6764 5    2021.02.10D19:02:25.332557500
4        2021.02.09D02:00:00.000000000 546.6978 4    2021.02.10D19:02:25.332557500
4        2021.02.09D03:00:00.000000000 164.0931 4    2021.02.10D19:02:25.332557500
4        2021.02.09D04:00:00.000000000 663.1649 4    2021.02.10D19:02:25.332557500

Async:

q)f:{res::y} / Store only the payload in "res"
q)neg[h](`.kxi.qsql;args;`f;(0#`)!()) / Make the API call and invoke callback
q)5#res / Display only the first 5 records
sensorID readTS                        valFloat qual updateTS
----------------------------------------------------------------------------------
4        2021.02.09D00:00:00.000000000 283.0992 1    2021.02.10D19:02:25.332557500
4        2021.02.09D01:00:00.000000000 906.6764 5    2021.02.10D19:02:25.332557500
4        2021.02.09D02:00:00.000000000 546.6978 4    2021.02.10D19:02:25.332557500
4        2021.02.09D03:00:00.000000000 164.0931 4    2021.02.10D19:02:25.332557500
4        2021.02.09D04:00:00.000000000 663.1649 4    2021.02.10D19:02:25.332557500
curl -X POST\
    --header "Content-Type: application/json"\
    --data '{"startTS":"2000.01.01D", "endTS":"2099.01.01D", "region": "amer", "query": "select from trace where sensorID=4"}'
    "http://${URL}/ping"