Skip to content

Release Notes - kdb Insights Enterprise 1.2.0

Release for kdb Insights Enterprise.

Release Date

2022-09-02

Features

[NEW] Install and Packages

  • Upgrade support in the kxi command line tool for easier upgrades. Please take note of other Upgrade Notes below.

  • Additional checks in the kxi CLI when deleting an install. A namespace must also now be provided when deleting installs using kxi install delete.

  • Ability to list, add and delete custom versioned user code via packages to kdb Insights Enterprise via the CLI outlined here.

  • Support within packages for the definition of User Defined Functions (UDFs) written in Python or q for deployment to stream-processor or use in the Explore Window.

[NEW] Pipelines

  • MQTT QoS 0 Reader has been added.

  • Users can run Test deploys in the UI, which provide user feedback on issues when building or editing a pipeline. This includes visualization of the data at each node in the pipeline, and clear indications of which nodes failed.

  • CPU and Memory resources can now be configured from the User Interface to allow rightsizing of data pipelines.

  • Use of User Defined Functions as part of a pipeline.

  • Apply Schema is now available for use after the CSV decoder, so the optional schema parameters for the CSV node have been removed from the Pipelines UI. Users should take care to ensure they're parsing all fields in the Apply Schema node when it's being applied to CSV decoded data.

[NEW] Explore and Query

  • Significant speed improvements to Explore. Query environments are pre-allocated as part of Data/assemblies, and so switching between tabs, assemblies, or instances is now instant. Query environments must be enabled in the Database/Data Access screen, or in assembly configuration, for the assembly to be available in Explore.

  • Filtering and aggregation are now available on the Query tab.

  • There is no need to choose a database when using the Query or SQL options in the Explore window. Label filtering can be used to select particular databases. You can now read from tables with the same name and schema across multiple databases when using the Query or SQL tabs.

  • Import and use of User Defined Functions in pipelines and the Explore Window to allow running of custom code.

  • Console output can now be cleared by right-clicking in the Console and choosing "Clear".

  • The query section at the top of Explore tabs can now be collapsed for a larger code area.

[NEW] Databases

  • CPU and Memory resources can now be configured from the User Interface to allow rightsizing of storage and query services.

  • Redundant replicas possible for all query-path processes, enabling greater query uptime despite process, node, or availability-zone failure.

  • Ability for Data Access tiers (RDB, IDB, HDB) to share compute resources by using mountList within Assembly configuration rather than mountName.

  • Granular assembly deployment status showing the state of deploying components, and assembly deployment errors, are now surfaced in the UI. Mouse-over the assembly's status icon while it's deploying to see the breakdown.

  • Environmental variables can now be configured in the UI for Data Access and Storage when the Advanced view is enabled.

  • Code View for Schemas is now available in the UI. Users can edit (or copy/paste) schemas directly, rather than configuring through UI forms.

[NEW] Performance

  • Ability to pause an RT stream for a single subscriber to let it catch up with other actions before resuming, without losing any data.

  • Cluster overprovisioning support that can be configured to enable faster scaling, by reserving resources.

[NEW] Java SDK

  • Query support has been added to the Java SDK.

  • For ingestion there is an option to validate the data being ingested to ensure it is in the format that is expected by the database, before it is ingested.

  • Batch publishing is available, if required.

[NEW] Observability

  • It is now possible to filter for a displayed value (for example, a particular Container) in the Diagnostics screen. Right-click on the value and select "Filter by [...]".

[NEW] Infrastructure

  • A storage class named sharedfiles, that is backed by shared storage, is automatically created when deploying via client Terraform scripts. sharedfiles storage is used for storing Packages and required by Insights Platform.

Fixes

A large number of fixes have gone into insights 1.2.0. Known issues which have been resolved, along with some other highlights, are listed below.

[FIX] Pipelines

  • Pipelines no longer report an initial "Unresponsive" status in the UI when launching.

[FIX] Explore and Query

  • With the introduction of Query Environments, there is no longer an issue where PVCs are not removed upon assembly teardown due to active Explore tabs targetting the assembly.

  • SQL queries can now target splayed tables (reference data).

[FIX] Observability

  • All UI notifications now appear in the Diagnostics panel.

[FIX] UI

  • Redeploying Data (assembly) no longer adds extra nodes to the navigation tree.

Artifacts

type location
Infrastructure kxi-terraform-1.2.0.tgz
Platform insights-1.2.0.tgz
Operator kxi-operator-1.2.0.tgz
CLI kxicli-1.2.0-py3-none-any.whl
ODBC Driver kodbc 1.2.0
Java SDK java-sdk 1.2.0

Upgrade notes

Upgrading assemblies

When upgrading Insights, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Version 1.2.0 requires a custom storageClassName definition

The addition of support for packages introduces a new storageClassName defined as sharedfiles, this is defined by default in any deployments of kdb Insights Enterprise that rely on the KX provided Terraform scripts for deployment but must be added manually by a user as outlined here.

Alternatively, within the deployment configuration as outlined here, modify the storageClassName to an alternative supported RWX storage option for example rookcephfs.

Keycloak 18

The version of Keycloak used within Insights has been upgraded from Keycloak 16 to Keycloak 18.

Please follow the Keycloak 18 migration steps to migrate your Keycloak data prior to upgrading Insights.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if Insights Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into Insights. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data
  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.
sm:
  size: 1
  • Torn-down pipelines that include Insights DB or Insights Stream Writers cannot be directly restarted after having their resources removed. Upon redeploy, data will not be published downstream from the pipeline. A workaround is to rename the pipeline. Resources are removed by checking "Clear Pipeline State" during teardown in the UI, or by directly deleting the Kubernetes PVCs backing the pipeline. If either of these things are done, the pipeline must be renamed before relaunching it.

  • .qsp.read.fromDatabase cannot be run in a pipeline to query an Insights database. The following example code can be used as a workaround to query data (ex. reference data to be used within a pipeline).

onResponse:{[hdr;data]
if[0 <> hdr`ac;
    .sp.log.error ("Query failed host=%s, api=%s, code=%n, error=%s";hdr`gw; hdr`api; hdr`ac; hdr`ai)];
publish data
}
.qsp.onStart {
    h:hopen hsym `$.spenv.getReleaseName[],"-sg-gateway:5050";
    res: neg[h] (`.kxi.getData; (enlist `table)!(enlist `instruments); `onResponse; ()!());
}
.qsp.run .qsp.read.fromCallback[`publish] .qsp.write.toConsole[]
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can navigate re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 if you're upgrading from an earlier version, for notes about