Skip to content

Get meta

The getMeta API returns meta information on APIs defined in the DAPs and aggregation functions in the Aggs for all DAPs and Aggs in the system. Additionally, this API returns information on the assemblies, and schemas.

Get Meta Version 3

This documentation covers an alternate format for .kxi.getMeta responses. You can set version to 3 in the opts dictionary of the .kxi.getMeta call to use this format.

Info

Using getMeta v3 is preferred, as it natively supports User Defined Analytics (UDAs). Both getMeta and getMeta v2 exist but are deprecated going forward. Transitioning to getMeta v3 ensures access to the latest features and improvements.

Gateway URL

The GATEWAY variable below defines an IPC connection to the Service Gateway. For example, :insights-qe-gateway:5050 connects to the query environment gateway within an insights namespace.

args: `startTS`endTS`labels!(startTS; endTS; labels);
opts:enlist[`version]!enlist 3; / Specify version 3
GATEWAY (`.kxi.getMeta; args; `callback; opts)

When issuing an IPC request, synchronous requests return a tuple. The first element is the response header, and the second is the response data. Refer to the asynchronous requests section for how to use the callback parameter to get an asynchronous response.

Gateway URL

The $GATEWAY variable must point to your kdb Insights installation. For a microservice installation, this is the hostname of the installation using port 8080. For an enterprise installation, this is your $INSIGHTS_HOSTNAME with /servicegateway as the URL prefix.

curl -G $GATEWAY/v3/meta" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json" \
    -d "$(jq -n \
        '{
            startTS : "",
            endTS   : "",
            labels  : {}
        }' | jq -cr .)"

Parameters

name required type description
startTS no timestamp Inclusive start time of the period of interest. It is recommended not to use this field with .kxi.getMeta calls.
endTS no timestamp Exclusive end time of the period of interest. It is recommended not to use this field with .kxi.getMeta calls.
labels no object Specifies the relevant assemblies (shards) to target the query. This object is a map of label names to desired label values. When running an IPC-based query, the type of each label must be a symbol or a list of symbols. For a REST request, this can either be a string or an array of strings. It is recommended to use this field in every request to narrow the scope of the query and target the DAPs. For more information, refer to database label configuration for configuring labels.

Time range

For getMeta, only DAPs with purview completely within the startTS and endTS are returned.

Labels

The labels parameter allows you to query against DAPs that are part of assemblies matching that label. For example, if there are DAPs belonging to two assemblies, one with a region label of canada and another with a region label of europe, then a dictionary with a region value of canada returns only the canada data. Assembly label arguments are optional and query all assemblies when not specified. If multiple labels are provided with multiple values, the cross-product of all label combinations is used to select target assemblies.

Example

In this example, assume you want to retrieve meta information for anything related to the region called emea up until the current time.

Gateway URL

The GATEWAY variable below defines an IPC connection to the Service Gateway. For example, :insights-qe-gateway:5050 connects to the query environment gateway within an insights namespace.

args:()!()

GATEWAY (`.kxi.getMeta; args; `; enlist[`version]!enlist 3)

This returns a tuple. The first element is a response header, and the second is the response data. For more information, refer to the header reference section for the details on the returned response header.

Gateway URL

The $GATEWAY variable must point to your kdb Insights installation. For a microservice installation, this is the hostname of the installation using port 8080. For an enterprise installation, this is your $INSIGHTS_HOSTNAME with /servicegateway as the URL prefix.

curl -G "$GATEWAY/v3/meta" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json" \
    -H "Authorization: Bearer $INSIGHTS_TOKEN" \
    -d "{}"

Response

This API returns a dictionary with 6 keys:

key type description
rc table Table of available Resource Coordinators and version information.
dap table Table of distinct Data Access Processes (DAPs).
api table Table of APIs (available in DAPs and Aggregators).
assembly table Table of assemblies fronted by the Service Gateway.
schema table Table of schemas in assemblies fronted by the Service Gateway.

The schemas for each table are:

Resource coordinators

This describes the set of available RCs, as well as version information for each of the other four keys. The schema is as follows:

column type description
rc symbol RC name.
labels table Table of labels covered by this RC.
started timestamp Start time of the SG's RC process.
api long Version number of api field.
agg long Version number of agg field.
assembly long Version number of assembly field.
schema long Version number of schema field.

The labels cells are tables whose columns represent the labels of the underlying DAPs, and the rows represent each label combination offered by the DAPs registered to this RC.

The 'version number' refers to the number of times that the SG's RC has updated that field. The number increments whenever the appropriate resource (Agg for agg, DAP for api, assembly, and schema) connects, disconnects, or updates the field. These fields can be used across multiple calls to the getMeta API, to detect changes since the previous call. If the RC process restarts, version numbers reset to 0. The started timestamp helps confirm any restart.

DAPs

This describes the set of distinct DAPs. The schema is as follows:

column type description
assembly symbol Assembly name.
instance symbol Instance type (corresponds to DAP's assembly elements.dap.instances entry).
startTS timestamp Start time of the DAP (inclusive).
endTS timestamp End time of the DAP (exclusive).

Note

Only distinct DAPs are shown. if two or more DAPs share the same assembly, instance, startTS, and endTS, only one entry is returned.

API

This describes the set of APIs and what labels or GWs offer them. The schema is as follows:

column type description
api symbol Name of the API.
query symbol Name of the function that runs on the DAPs
aggregation symbol Name of the function that runs on the Aggregator, and consolidates the partial results from DAPs.
uda boolean Whether this is a User Defined Analytic (true) or a built-in KXI API (false).
description string Description of the API.
params dictionary[] Metadata for parameters of API. Can include keys such as name, type,isReq,description.
misc dictionary[] Miscellaneous metadata about the API.
procs table Table of individual DAP and Agg processes corresponding to this entry if not full.
labels table Which label value combinations support this API.
full boolean True if all DAPs or Aggs for these labels support this API and agg function mapping.

If full is true, the procs cell is empty. Otherwise, procs contains a table showing all the DAPs/RCs/Aggs to which this API and agg function mapping apply. This may be used to define preferential routing to access specific versions of the API, or determine which labels to target in API arguments. The schema of the table is as follows:

Old registration

If old registration methods, such as .da.registerAPI or .sgagg.registerAggFn, were used to register in either the DAPs or aggregators, the system does not attempt to fill out the procs table when full is false. You can confirm this by checking for the _oldRegistration key under the misc column for this API.

column type description
dap symbol Host/port of DAP.
agg symbol Host/port of Agg.
rc symbol Name of RC of DAP/Agg.
labels symbol Value of label_1 of DAP.

Assembly

This describes the assemblies fronted by the Service Gateway:

column type description
assembly symbol Assembly name.
labels dictionary Labels for this assembly.
tbls symbol[] List of tables in the assembly.

Schema

This describes the tables contained in the assemblies fronted by the Service Gateway:

column type description
table symbol Table name.
assembly symbol[] Assemblies that contain this table with this schema.
typ symbol Table type.
pkCols symbol[] Primary key columns.
prtnCol symbol Column the data is partitioned on.
sortColsMem symbol[] Columns the data is sorted on in memory (e.g. RDB).
sortColsIDisk symbol[] Columns the data is sorted on intraday disk (e.g. IDB).
sortColsDisk symbol[] Columns the data is sorted on on-disk (e.g. HDB).
isSplayed boolean Indicates if the table is splayed.
isPartitioned boolean Indicates if the table is partitioned.
isSharded boolean Indicates if the table is sharded.
description string Table description.
columns table Table columns (refer to the table below).

Each columns cell is a table with the following columns:

column type description example
column symbol Column name. `price
description string Column description. "Trade price"
typ short KDB+ type. 9h
attrMem symbol In-memory attribute (e.g. RDB). `
attrIDisk symbol On intraday disk attribute (e.g. IDB). `s
attrDisk symbol On-disk attribute (e.g. HDB). `p
isSerialized boolean Whether column is serialized. 0b
fk symbol Foreign table information. `instrument.sym

A table may appear in multiple rows if it appears in different assemblies with different schema or properties. For example:

table assembly typ        ..
--------------------------..
trade ,`asm1   partitioned..
trade ,`asm2   splayed    ..
..

Example:

Assume in the following example that assemblies are labeled by region with three regions: amer`apac`emea.

args:()!()
res:last h(`.kxi.getMeta;args;`;(0#`)!())
res`rc
api agg assembly schema rc      labels                   started
--------------------------------------------------------------------------------------
6   1   2        2      sg-rc-0 +(,`region)!,`amer`apac  2024.08.27D21:17:17.123353833
3   1   1        1      sg-rc-1 +(,`region)!,,`emea      2024.08.27D21:26:44.049902785
res`dap
assembly  instance startTS                       endTS
------------------------------------------------------------------------------
amer      rdb      2024.08.27D21:25:45.000000000 0W
amer      hdb      -0W                           2024.08.27D21:22:15.000000001
amer      idb      2024.08.27D21:22:15.000000001 2024.08.27D21:25:45.000000000
apac      hdb      -0W                           2024.08.27D21:23:07.000000001
apac      idb      2024.08.27D21:23:07.000000001 2024.08.27D21:26:12.000000000
apac      rdb      2024.08.27D21:26:12.000000000 0W
emea      rdb      2024.08.27D21:26:39.000000000 0W
emea      idb      2024.08.27D21:23:02.000000001 2024.08.27D21:26:39.000000000
emea      hdb      -0W                           2024.08.27D21:23:02.000000001
-3?res`api / Sample 3 of the APIs
api          query        aggregation    uda  description                                                   params                                                                                 ..
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------..
.kxi.ping    .kxi.ping                   0    "Returns true if it reaches a target process."                ()                                                                                     ..
.kxi.getData .kxi.getData .sgagg.getData 0    "Get timeseries and splayed data from data access processes." (`name`type`isReq`description!(`table;-11h;1b;"Table to select from.");`name`type`isReq..
.kxi.preview .kxi.preview .sgagg.preview 0    "Efficiently fetch a small preview of a table."               (`name`type`isReq`description!(`table;-11h;1b;"Table to preview.");`name`type`isReq`def..
exec from res`api where api=`.kxi.getData / View single record
api        | `.kxi.getData
query      | `.kxi.getData
aggregation| `.sgagg.getData
uda        | 0b
description| "Get timeseries and splayed data from data access processes."
params     | (`name`type`isReq`description!(`table;-11h;1b;"Table to select from.");`name`type`isReq`description!(`startTS;-12h;0b;"Inclusive start time for period of interest.");`name`type`isReq`d..
return     | `type`description!(99h;"Table records with parameters applied")
misc       | (`symbol$())!()
procs      | ()
labels     | +(,`region)!,`amer`apac`emea
full       | 1b
res`assembly / Assume one assembly per region
assembly  labels              tbls
--------------------------------------------------------------------
amer      (,`region)!,`amer   trade quote exchange instrument aTable
apac      (,`region)!,`apac   trade quote exchange instrument aTable
apac      (,`region)!,`apac   trade quote exchange instrument aTable
res`schema
table      assembly                typ         pkCols        prtnCol  sortColsMem sortColsIDisk sortColsDisk isSplayed isPartitioned isSharded description                                     column..
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------..
aTable     amer apac apac        splayed_mem `symbol$()                         aSymbol       `symbol$()   1         0             0         "Case sensitive"                                (+(,`c..
exchange   amer apac apac        basic       `symbol$()                         exchangeID    `symbol$()   0         0             0         "Money can be exchanged for goods and services" (+(,`c..
instrument amer apac apac        splayed     ,`sym                              sym           `symbol$()   1         0             1         "Pianos, recorders, drums"                      (+(,`c..
quote      amer apac apac        partitioned `symbol$()    realTime             sym           ,`sym        1         1             1         "Quote data"                                    (+(,`c..
trade      amer apac apac        partitioned `sym`realTime realTime             sym           ,`sym        1         1             1         "Trade data"                                    (+(,`c..
last res`schema / View single record
table        | `trade
assembly     | `amer`apac`apac
typ          | `partitioned
pkCols       | `sym`realTime
prtnCol      | `realTime
sortColsMem  | `symbol$()
sortColsIDisk| ,`sym
sortColsDisk | ,`sym
isSplayed    | 1b
isPartitioned| 1b
isSharded    | 1b
description  | "Trade data"
columns      | (+(,`column)!,`time`sym`realTime`price`size)!+`typ`description`oldName`attrMem`attrIDisk`attrDisk`isSerialized`foreign`anymap`backfill`encrypt`vectorIndex!(-16 -11 -12 -9 -7h;("Time ..
res[`schema;4;`columns] / View columns for trade table
column  | typ description   oldName attrMem attrIDisk attrDisk isSerialized foreign anymap backfill encrypt
--------| -------------------------------------------------------------------------------------------------
time    | -16 "Time of day"                                    0                    0               0
sym     | -11 "Symbol name"         g       p         p        0                    0               0
realTime| -12 "Timestamp"                                      0                    0               0
price   | -9  "Trade price"                                    0                    0               0
size    | -7  "Trade size"                                     0                    0               0

Troubleshooting

For issues with the query or responses, refer to the troubleshooting guide.