Upgrading

As new features are added to the kdb Insights Database, APIs and state may change. This document outlines considerations when upgrading between different versions of the kdb Insights Database.

Upgrade Compatibility

For changes that would result in a breaking change, an upwards compatible path will always be supported in minor release versions. Features may be deprecated in a minor release and could be removed in a major release upgrade.

Upgrading to 1.5

Labels in 1.5

Querying with labels has changed in 1.5 to be a distinguished top level parameter. Previously, labels could be included as directly as parameters in requests to query APIs such as getData or sql. This has been changed such that labels are now nested under a labels object or prefixed with label_ in SQL. This change has been made to resolve an issue of unresolvable collisions between labels and custom API parameters or table columns. For example, previously if you had a table with a column called region and an assembly with a label called region, referencing region in a query was ambiguous and could result in undesired results. With this change, region in a query will always refer to the table column and a labels object is used to refer to the label called region.

Deprecation notice

In the 1.5 release, the old label style is still supported but is deprecated and will result in a warning log in the Resource Coordinator. An extra environment variable ALLOW_OLD_LABEL_STYLE has been added to the Resource Coordinator that preserves the old behaviour prior to 1.5 which defaults to "true". While enabled, this allows both the old and new label parameter style in the same query. In 1.6, this will now default to "false" but can be enabled by overriding the environment variable setting. In 2.0, this feature will be removed entirely.

Upgrading

A slight modification is required to convert queries to the new format. Requests now must specify labels as a top level parameter for getData requests or custom APIs. For SQL requests, labels in the query must have the label_ prefix.

Get Data

For full API details, see the getData reference page.

qREST

Gateway URL

The GATEWAY variable below is defined as an IPC connection to the Service Gateway. For example `:insights-qe-gateway:5050 would connect to the query environment gateway within an insights namespace.

Prior to 1.5, the region label in a request would be top level in the argument field.

args: (!) . flip (
    (`table   ; `trace);
    (`region  ; `$"us-east-1");
    (`startTS ; .z.p - 0D00:05:00); // Select the last 5 minutes of data
    (`endTS   ; .z.p)
    )

GATEWAY (`.kxi.getData; args; `; ()!())

In 1.5, this should now be nested under a labels object.

args: (!) . flip (
    (`table   ; `trace);
    (`labels  ; enlist[`region]!enlist`$"us-east-1");
    (`startTS ; .z.p - 0D00:05:00); // Select the last 5 minutes of data
    (`endTS   ; .z.p)
    )

GATEWAY (`.kxi.getData; args; `; ()!())

Gateway URL

The $GATEWAY variable should point at your kdb Insights install. For a microservice install, this will be the hostname of the install using port 8080. For an enterprise install, this is your $INSIGHTS_HOSTNAME with /servicegateway as the URL prefix.

Prior to 1.5, the region label in a request would be top level in the argument field.

curl -X POST "$GATEWAY/kxi/getData" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json" \
    -H "Authorization: Bearer $INSIGHTS_TOKEN" \
    -d "$(jq -n \
        --arg startTS "$(date -u '+%Y.%m.%dD%H:00:00')" \
        --arg endTS "$(date -u '+%Y.%m.%dD%H:%M%:%S')" \
        '{
            table   : "trace",
            region  : "us-east-1",
            startTS : $startTS,
            endTS   : $endTS
        }' | jq -cr .)"

In 1.5, this should now be nested under a labels object.

curl -X POST "$GATEWAY/kxi/getData" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json" \
    -H "Authorization: Bearer $INSIGHTS_TOKEN" \
    -d "$(jq -n \
        --arg startTS "$(date -u '+%Y.%m.%dD%H:00:00')" \
        --arg endTS "$(date -u '+%Y.%m.%dD%H:%M%:%S')" \
        '{
            table   : "trace",
            labels  : { region: "us-east-1" },
            startTS : $startTS,
            endTS   : $endTS
        }' | jq -cr .)"

SQL

For full API details, see the sql reference page.

qREST

Gateway URL

The GATEWAY variable below is defined as an IPC connection to the Service Gateway. For example `:insights-qe-gateway:5050 would connect to the query environment gateway within an insights namespace.

Prior to 1.5, the exchange label in a query would be referenced directly.

query: "select date,sym,avg(price) from trade ",
  "where (date between '2021.01.01' and '2021.01.07') ",
  "and (exchange='nyse') group by date,sym";
GATEWAY (`.kxi.sql; enlist[`query]!enlist query;`;()!())

In 1.5, this should now have a label_ prefix.

query: "select date,sym,avg(price) from trade ",
  "where (date between '2021.01.01' and '2021.01.07') ",
  "and (label_exchange='nyse') group by date,sym";
GATEWAY (`.kxi.sql; enlist[`query]!enlist query;`;()!())

Gateway URL

The $GATEWAY variable should point at your kdb Insights install. For a microservice install, this will be the hostname of the install using port 8080. For an enterprise install, this is your $INSIGHTS_HOSTNAME with /servicegateway/qe as the URL prefix.

Prior to 1.5, the exchange label in a query would be referenced directly.

select date,sym,avg(price) from trade
    where (date between '2021.01.01' and '2021.01.07') and (exchange='nyse')
    group by date,sym

In 1.5, this should now have a label_ prefix.

select date,sym,avg(price) from trade
    where (date between '2021.01.01' and '2021.01.07') and (label_exchange='nyse')
    group by date,sym

This example uses the above query set as a variable called $QUERY.

curl -X POST "$GATEWAY/kxi/sql" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json" \
    -H "Authorization: Bearer $INSIGHTS_TOKEN" \
    -d "$(jq -n --arg query "$QUERY" '{ query: $query }' | jq -cr .)"