Get meta
The getMeta API returns meta information on APIs defined in the DAPs and aggregation functions in the Aggs for all DAPs and Aggs in the system. Additionally, this API returns information on the assemblies, and schemas.
Get Meta Version 3
This documentation covers an alternate format for .kxi.getMeta responses. You can set version to 3 in the opts dictionary of the .kxi.getMeta call to use this format.
Info
Using getMeta v3 is preferred, as it natively supports User Defined Analytics (UDAs). Both getMeta and getMeta v2 exist but are deprecated going forward. Transitioning to getMeta v3 ensures access to the latest features and improvements.
Gateway URL
The GATEWAY variable below defines an IPC connection to the Service Gateway. For example, :insights-qe-gateway:5050 connects to the query environment gateway within an insights namespace.
args: `startTS`endTS`labels!(startTS; endTS; labels);
opts:enlist[`version]!enlist 3; / Specify version 3
GATEWAY (`.kxi.getMeta; args; `callback; opts)
When issuing an IPC request, synchronous requests return a tuple. The first element is the response header, and the second is the response data. Refer to the asynchronous requests section for how to use the callback parameter to get an asynchronous response.
Gateway URL
The $GATEWAY variable must point to your kdb Insights installation. For a microservice installation, this is the hostname of the installation using port 8080. For an enterprise installation, this is your $INSIGHTS_HOSTNAME with /servicegateway as the URL prefix.
curl -G $GATEWAY/v3/meta" \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-d "$(jq -n \
'{
startTS : "",
endTS : "",
labels : {}
}' | jq -cr .)"
Parameters
| name | required | type | description |
|---|---|---|---|
startTS |
no | timestamp |
Inclusive start time of the period of interest. It is recommended not to use this field with .kxi.getMeta calls. |
endTS |
no | timestamp |
Exclusive end time of the period of interest. It is recommended not to use this field with .kxi.getMeta calls. |
labels |
no | object |
Specifies the relevant assemblies (shards) to target the query. This object is a map of label names to desired label values. When running an IPC-based query, the type of each label must be a symbol or a list of symbols. For a REST request, this can either be a string or an array of strings. It is recommended to use this field in every request to narrow the scope of the query and target the DAPs. For more information, refer to database label configuration for configuring labels. |
Time range
For getMeta, only DAPs with purview completely within the startTS and endTS are returned.
Labels
The labels parameter allows you to query against DAPs that are part of assemblies matching that label. For example, if there are DAPs belonging to two assemblies, one with a region label of canada and another with a region label of europe, then a dictionary with a region value of canada returns only the canada data. Assembly label arguments are optional and query all assemblies when not specified. If multiple labels are provided with multiple values, the cross-product of all label combinations is used to select target assemblies.
Example
In this example, assume you want to retrieve meta information for anything related to the region called emea up until the current time.
Gateway URL
The GATEWAY variable below defines an IPC connection to the Service Gateway. For example, :insights-qe-gateway:5050 connects to the query environment gateway within an insights namespace.
args:()!()
GATEWAY (`.kxi.getMeta; args; `; enlist[`version]!enlist 3)
This returns a tuple. The first element is a response header, and the second is the response data. For more information, refer to the header reference section for the details on the returned response header.
Gateway URL
The $GATEWAY variable must point to your kdb Insights installation. For a microservice installation, this is the hostname of the installation using port 8080. For an enterprise installation, this is your $INSIGHTS_HOSTNAME with /servicegateway as the URL prefix.
curl -G "$GATEWAY/v3/meta" \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer $INSIGHTS_TOKEN" \
-d "{}"
Response
This API returns a dictionary with 6 keys:
| key | type | description |
|---|---|---|
rc |
table | Table of available Resource Coordinators and version information. |
dap |
table | Table of distinct Data Access Processes (DAPs). |
api |
table | Table of APIs (available in DAPs and Aggregators). |
assembly |
table | Table of assemblies fronted by the Service Gateway. |
schema |
table | Table of schemas in assemblies fronted by the Service Gateway. |
The schemas for each table are:
Resource coordinators
This describes the set of available RCs, as well as version information for each of the other four keys. The schema is as follows:
| column | type | description |
|---|---|---|
rc |
symbol | RC name. |
labels |
table | Table of labels covered by this RC. |
started |
timestamp | Start time of the SG's RC process. |
api |
long | Version number of api field. |
agg |
long | Version number of agg field. |
assembly |
long | Version number of assembly field. |
schema |
long | Version number of schema field. |
The labels cells are tables whose columns represent the labels of the underlying DAPs, and the rows represent each label combination offered by the DAPs registered to this RC.
The 'version number' refers to the number of times that the SG's RC has updated that field. The number increments whenever the appropriate resource (Agg for agg, DAP for api, assembly, and schema) connects, disconnects, or updates the field. These fields can be used across multiple calls to the getMeta API, to detect changes since the previous call. If the RC process restarts, version numbers reset to 0. The started timestamp helps confirm any restart.
DAPs
This describes the set of distinct DAPs. The schema is as follows:
| column | type | description |
|---|---|---|
assembly |
symbol | Assembly name. |
instance |
symbol | Instance type (corresponds to DAP's assembly elements.dap.instances entry). |
startTS |
timestamp | Start time of the DAP (inclusive). |
endTS |
timestamp | End time of the DAP (exclusive). |
Note
Only distinct DAPs are shown. if two or more DAPs share the same assembly, instance, startTS, and endTS, only one entry is returned.
API
This describes the set of APIs and what labels or GWs offer them. The schema is as follows:
| column | type | description |
|---|---|---|
api |
symbol | Name of the API. |
query |
symbol | Name of the function that runs on the DAPs |
aggregation |
symbol | Name of the function that runs on the Aggregator, and consolidates the partial results from DAPs. |
uda |
boolean | Whether this is a User Defined Analytic (true) or a built-in KXI API (false). |
description |
string | Description of the API. |
params |
dictionary[] | Metadata for parameters of API. Can include keys such as name, type,isReq,description. |
misc |
dictionary[] | Miscellaneous metadata about the API. |
procs |
table | Table of individual DAP and Agg processes corresponding to this entry if not full. |
labels |
table | Which label value combinations support this API. |
full |
boolean | True if all DAPs or Aggs for these labels support this API and agg function mapping. |
If full is true, the procs cell is empty. Otherwise, procs contains a table showing all the DAPs/RCs/Aggs to which this API and agg function mapping apply. This may be used to define preferential routing to access specific versions of the API, or determine which labels to target in API arguments. The schema of the table is as follows:
Old registration
If old registration methods, such as .da.registerAPI or .sgagg.registerAggFn, were used to register in either the DAPs or aggregators, the system does not attempt to fill out the procs table when full is false. You can confirm this by checking for the _oldRegistration key under the misc column for this API.
| column | type | description |
|---|---|---|
dap |
symbol | Host/port of DAP. |
agg |
symbol | Host/port of Agg. |
rc |
symbol | Name of RC of DAP/Agg. |
labels |
symbol | Value of label_1 of DAP. |
Assembly
This describes the assemblies fronted by the Service Gateway:
| column | type | description |
|---|---|---|
assembly |
symbol | Assembly name. |
labels |
dictionary | Labels for this assembly. |
tbls |
symbol[] | List of tables in the assembly. |
Schema
This describes the tables contained in the assemblies fronted by the Service Gateway:
| column | type | description |
|---|---|---|
table |
symbol | Table name. |
assembly |
symbol[] | Assemblies that contain this table with this schema. |
typ |
symbol | Table type. |
pkCols |
symbol[] | Primary key columns. |
prtnCol |
symbol | Column the data is partitioned on. |
sortColsMem |
symbol[] | Columns the data is sorted on in memory (e.g. RDB). |
sortColsIDisk |
symbol[] | Columns the data is sorted on intraday disk (e.g. IDB). |
sortColsDisk |
symbol[] | Columns the data is sorted on on-disk (e.g. HDB). |
isSplayed |
boolean | Indicates if the table is splayed. |
isPartitioned |
boolean | Indicates if the table is partitioned. |
isSharded |
boolean | Indicates if the table is sharded. |
description |
string | Table description. |
columns |
table | Table columns (refer to the table below). |
Each columns cell is a table with the following columns:
| column | type | description | example |
|---|---|---|---|
column |
symbol | Column name. | `price |
description |
string | Column description. | "Trade price" |
typ |
short | KDB+ type. | 9h |
attrMem |
symbol | In-memory attribute (e.g. RDB). | ` |
attrIDisk |
symbol | On intraday disk attribute (e.g. IDB). | `s |
attrDisk |
symbol | On-disk attribute (e.g. HDB). | `p |
isSerialized |
boolean | Whether column is serialized. | 0b |
fk |
symbol | Foreign table information. | `instrument.sym |
A table may appear in multiple rows if it appears in different assemblies with different schema or properties. For example:
table assembly typ ..
--------------------------..
trade ,`asm1 partitioned..
trade ,`asm2 splayed ..
..
Example:
Assume in the following example that assemblies are labeled by region with three regions: amer`apac`emea.
args:()!()
res:last h(`.kxi.getMeta;args;`;(0#`)!())
res`rc
api agg assembly schema rc labels started
--------------------------------------------------------------------------------------
6 1 2 2 sg-rc-0 +(,`region)!,`amer`apac 2024.08.27D21:17:17.123353833
3 1 1 1 sg-rc-1 +(,`region)!,,`emea 2024.08.27D21:26:44.049902785
res`dap
assembly instance startTS endTS
------------------------------------------------------------------------------
amer rdb 2024.08.27D21:25:45.000000000 0W
amer hdb -0W 2024.08.27D21:22:15.000000001
amer idb 2024.08.27D21:22:15.000000001 2024.08.27D21:25:45.000000000
apac hdb -0W 2024.08.27D21:23:07.000000001
apac idb 2024.08.27D21:23:07.000000001 2024.08.27D21:26:12.000000000
apac rdb 2024.08.27D21:26:12.000000000 0W
emea rdb 2024.08.27D21:26:39.000000000 0W
emea idb 2024.08.27D21:23:02.000000001 2024.08.27D21:26:39.000000000
emea hdb -0W 2024.08.27D21:23:02.000000001
-3?res`api / Sample 3 of the APIs
api query aggregation uda description params ..
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------..
.kxi.ping .kxi.ping 0 "Returns true if it reaches a target process." () ..
.kxi.getData .kxi.getData .sgagg.getData 0 "Get timeseries and splayed data from data access processes." (`name`type`isReq`description!(`table;-11h;1b;"Table to select from.");`name`type`isReq..
.kxi.preview .kxi.preview .sgagg.preview 0 "Efficiently fetch a small preview of a table." (`name`type`isReq`description!(`table;-11h;1b;"Table to preview.");`name`type`isReq`def..
exec from res`api where api=`.kxi.getData / View single record
api | `.kxi.getData
query | `.kxi.getData
aggregation| `.sgagg.getData
uda | 0b
description| "Get timeseries and splayed data from data access processes."
params | (`name`type`isReq`description!(`table;-11h;1b;"Table to select from.");`name`type`isReq`description!(`startTS;-12h;0b;"Inclusive start time for period of interest.");`name`type`isReq`d..
return | `type`description!(99h;"Table records with parameters applied")
misc | (`symbol$())!()
procs | ()
labels | +(,`region)!,`amer`apac`emea
full | 1b
res`assembly / Assume one assembly per region
assembly labels tbls
--------------------------------------------------------------------
amer (,`region)!,`amer trade quote exchange instrument aTable
apac (,`region)!,`apac trade quote exchange instrument aTable
apac (,`region)!,`apac trade quote exchange instrument aTable
res`schema
table assembly typ pkCols prtnCol sortColsMem sortColsIDisk sortColsDisk isSplayed isPartitioned isSharded description column..
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------..
aTable amer apac apac splayed_mem `symbol$() aSymbol `symbol$() 1 0 0 "Case sensitive" (+(,`c..
exchange amer apac apac basic `symbol$() exchangeID `symbol$() 0 0 0 "Money can be exchanged for goods and services" (+(,`c..
instrument amer apac apac splayed ,`sym sym `symbol$() 1 0 1 "Pianos, recorders, drums" (+(,`c..
quote amer apac apac partitioned `symbol$() realTime sym ,`sym 1 1 1 "Quote data" (+(,`c..
trade amer apac apac partitioned `sym`realTime realTime sym ,`sym 1 1 1 "Trade data" (+(,`c..
last res`schema / View single record
table | `trade
assembly | `amer`apac`apac
typ | `partitioned
pkCols | `sym`realTime
prtnCol | `realTime
sortColsMem | `symbol$()
sortColsIDisk| ,`sym
sortColsDisk | ,`sym
isSplayed | 1b
isPartitioned| 1b
isSharded | 1b
description | "Trade data"
columns | (+(,`column)!,`time`sym`realTime`price`size)!+`typ`description`oldName`attrMem`attrIDisk`attrDisk`isSerialized`foreign`anymap`backfill`encrypt`vectorIndex!(-16 -11 -12 -9 -7h;("Time ..
res[`schema;4;`columns] / View columns for trade table
column | typ description oldName attrMem attrIDisk attrDisk isSerialized foreign anymap backfill encrypt
--------| -------------------------------------------------------------------------------------------------
time | -16 "Time of day" 0 0 0
sym | -11 "Symbol name" g p p 0 0 0
realTime| -12 "Timestamp" 0 0 0
price | -9 "Trade price" 0 0 0
size | -7 "Trade size" 0 0 0
Troubleshooting
For issues with the query or responses, refer to the troubleshooting guide.