Skip to content

Release Notes - kdb Insights Enterprise 1.4.0

Release for kdb Insights Enterprise.

Release Date

2023-03-03

New Features

Introducing Pythonic pipeline development in the UI, dynamic logging notifications providing direct access to root cause of underlying issues, an enhanced user experience for database configuration and additional security enhancements.

[NEW] Pipelines

  • You can now use Python in the pipeline builder in the UI to perform aggregations and analytics.

  • Ingest large batches of data directly into the historical database has been moved out of beta.

    • Use fewer resources. Memory and disk resources greatly reduced during ingest.
    • Select the Write Direct to HDB field in the Database Writer in the UI, or use the q and Python APIs.

[NEW] User Interface

  • Improvements to the observability user experience with two new icons in top right hand-side of the UI banner:

    • A notification icon shows a count of new log messages. This ensures you are notified of any new error messages wherever you are in the UI. You can also click on the icon to show more details of the message, these are displayed in the right-hand third of the screen.

    • A settings icon enables you to update the application logging level for future log records written by the kdb Insights Enterprise components.

  • You can now share the URL of a View by selecting the view on the left-hand toolbar, right-clicking and choosing the new Share button. This URL just shares the content of the View and not the toolbars (e.g. it does not show the left-hand toolbar or the Design/Preview panel at the top).

[NEW] Databases

A new version of our Database Configuration page is now available, under a beta flag, that provides an enhanced user experience. It can be toggled on and off from the Settings option in the main toolbar. We recommend that you use the existing Database Configuration age for all running databases, but please try the new user interface out and give us your feedback.

[NEW] Packages

The user interface now allows you to configure which Packages and their associated Custom APIs are used within individual databases, this had previously only been possible through kdb CLI deployments.

[NEW] Security Improvements

  • Keycloak has been updated to version 19. This impacts the following:
    • The Keycloak Admin console has been changed
    • Logout screen provides a redirection to the login page

Improvements

Keycloak realm

In this release, the handling of the Keycloak realm has been improved. This realm contains the role definitions, users and clients. Previously the realm was not being imported by default on install. The role definitions will always be imported as part of an install or upgrade now. The users and clients will only be imported by default on install (kxi install), and not on upgrade.

To make changes to the initClient and initUser on upgrade, you will need to tell the application to import them using the setting below.

keycloak:
  importUsers: true

Reliable Transport

The kdb Insights Enterprise transport layer has adopted a version of Raft called NuRaft. This improves the fault tolerance and performance of kdb Insights Enterprise. NuRaft is an open source version of Raft. Migrating to NuRaft also provides kdb Insights Enterprise with additional capabilities for future enhancements to Reliable Transport.

Third-party dependencies

kdb Insights Enterprise 1.4.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.4.0.tgz
Operator kxi-operator-1.4.0.tgz
CLI kxicli-1.4.0-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database - this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • If end-of-day processing is interrupted and remains incomplete until the next EOD period, this prevents further EODs from completing and requires manual recovery.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored. 

"[] Failed to compare saved assembly  to deployed version, error: type"

  • When creating a pipeline the user interface does not stop you from creating it with the same name as an existing pipeline, however two pipelines with the same name may not be deploy correctly. Please ensure all pipelines have different names.

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml. 

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

Backward Compatibility

If you are upgrading from a version earlier than 1.3.3 you must also read each of the relevant release notes to ensure you know all the changes that will affect the upgrade.

The most pertinent upgrade notes from previous versions are listed below: