Skip to content

Release Notes - kdb Insights Enterprise 1.4.2

Release for kdb Insights Enterprise.

Release Date

2023-03-17

Fixes

[FIX] Resource utilization in the UI Scratchpad has been greatly improved. This fix along with a larger default memory allocation provide greater Scratchpad process stability.

Third-party dependencies

kdb Insights Enterprise 1.4.2 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.4.2.tgz
Operator kxi-operator-1.4.0.tgz
CLI kxicli-1.4.1-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database - this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored. 

"[] Failed to compare saved assembly  to deployed version, error: type"

  • When creating a pipeline the user interface does not stop you from creating it with the same name as an existing pipeline, however two pipelines with the same name may not be deploy correctly. Please ensure all pipelines have different names.

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml. 

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

Backward Compatibility

If you are upgrading from a version earlier than 1.3.3 you must also read each of the relevant release notes to ensure you know all the changes that will affect the upgrade.

The most pertinent upgrade notes from previous versions are listed below: