Skip to content

Release Notes - kdb Insights Enterprise 1.3.0

Release for kdb Insights Enterprise.

Release Date

2022-10-26

Features

[NEW] User Interface

  • Items in the page tree have been renamed for clarity and consistency with the CLI.
  • Data is now known as Assemblies
  • Explore is now known as Queries
  • Reports is now known as Views

[NEW] Pipelines

  • An "Auto" setting for Parse Strings in the Apply Schema node has been added. The column will be parsed if the input is textual (strings or bytes), but the output type is non-textual.

  • Data types are now visible for each column for Test Deploy previews. These can be used to validate the data types and the data shape of Transforms and Maps results.

  • Data types and column names are validated for KX Insights Database writers as part of a test deploy.

  • Additional Reader validation has been introduced to pipelines, to ensure correctness of settings before deploying.

  • Test deploy previews can now show non-tabular data.

  • Previews are now available for Reader nodes, before data has been decoded.

  • The visibility of a Pipeline's state has been improved in the Overview tab. Warning and error counts can be clicked on to launch a diagnostics tab for that Pipeline.

[NEW] Query

  • Environments in the Query tab are now per-user - variables & functions used in the Scratchpad will no longer clash with other users'.

[NEW] Assemblies

  • It is now possible to see assemblies created using kxi assembly create in the UI. Note that assemblies created and managed through the command-line are not editable via the UI.

  • Additional logging has been added to the Storage Manager, to ease troubleshooting.

  • Schema configuration validation has been improved.

[NEW] Install and Packages

  • The KXI Operator is no longer uninstalled by default by kxi install delete, the command line parameter --uninstall-operator must now be used.

  • Many values for kxi install setup can now be passed via command line parameter, please see the CLI Reference for more detail.

  • Improved support for rolling back to previous versions. Enter a previous version number using the upgrade command kxi install upgrade to roll back.

  • The current kubectl context is now used as the default in all commands, instead of the namespace specified in the cli-config file.

  • kdb Insights Python modules for packaging and machine learning are no longer dependent on libc.musl, and can now be used for development on Windows and Mac.

  • Custom user query APIs and Aggregation functions can be installed and initialised within Data Access Processes and Aggregators as outlined here.

[NEW] Observability

  • Reliable Transport diagnostics are now available in diagnostics exports from the UI.

[NEW] Infrastructure

  • The base image for most components has been moved to Rocky Linux.

Fixes

A large number of fixes have gone into kdb Insights Enterprise 1.3.0. Known issues which have been resolved, along with some other highlights, are listed below.

[FIX] Streams and Reliable Transport

  • A hard reset can now be performed on Reliable Transport, to recover from stream corruption. Please contact KX Support for usage instructions.

[FIX] Pipelines

  • Pipelines which have their data cleared when they're torn-down can now be re-deployed with the same name. Previously, this would cause RT sequence number problems, stopping data from flowing.

  • Persistent pop-up errors in the UI after resizing a Map node have been resolved.

  • .qsp.read.fromDatabase can now be used in pipelines.

  • Renaming a pipeline node no longer disconnects its edges.

  • Empty pipelines are no longer deployable.

[FIX] User Interface

  • Validation fixes so that date and time fields can be entered without use of the picker.

[FIX] Java SDK

  • Occasional authentication failures when querying via Java SDK have been resolved. These would previously be seen more frequently on slower connections.

Artifacts

type location
Infrastructure kxi-terraform-1.3.0.tgz
Platform insights-1.3.0.tgz
Operator kxi-operator-1.3.0.tgz
CLI kxicli-1.3.0-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the Insights values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of Insights Platform.

Users who currently set initUser.enabled=true or initClient.enabled=true within their own values.yaml may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
    .Values.keycloak.initUser.name
    .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
    .Values.keycloak.initClient.clientId
    .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"
Where keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, the user is required to set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here

Assembly blockSize changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

  • if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
    • this has the highest performance, but has no RAM limit on received data
  • if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
    • the smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below 
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

  • The user-defined variables & functions in the Queries tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • The Queries tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Queries tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1
  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 and kdb Insights Enterprise 1.2.0 if you're upgrading from versions earlier than 1.1.0 or 1.2.0, for notes about