Skip to content

Release Notes - kdb Insights Enterprise 1.3.3

Release for kdb Insights Enterprise.

Release Date

2023-01-17

New Features

Functionality for improved ingestion, efficiency and reduction of resources. Take advantage of an IDE to develop your databases, pipelines and query interfaces.

[NEW] Improved Ingestion

  • [Beta] Ingest large batches of data directly into the historical database:

    • Use fewer resources. Memory and disk resources greatly reduced during ingest.
    • Select the Direct Writing field in the Database Writer in the UI, or use the q and Python APIs.

    See details here for known issues.

[NEW] Writing data to Amazon S3

  • You can now write data to Amazon S3 by either:

[NEW] Improved IDE Capabilities

  • Build YAML files and upload using them kdb Insights CLI:

    • Speed up development workflow, improve productivity.
    • Access to the Kubernetes control plane is no longer needed. This is now handled for you by the CLI.
    • Assembly Management using CLI
  • If you have access to KX Analyst, you can use it to develop structured functions. You can now:

    • Enjoy a rich IDE development environment with syntax highlighting for YAML files.
    • Query kdb Insights Enterprise data using REST API or query interface.
    • Upload using the CLI or as part of CI/CD pipeline.
    • Build a complete SDLC that harnesses the flexibility of KX Analyst against kdb Insights Enterprise data.

    Note

    Contact your KX Sales representative if you are interested in using KX Analyst. In future releases additional IDEs will become available including VS Code and Jupyter Notebooks.

Improvements

[NEW] Reliable Transport

  • Reliable Transport's logging default level for the sequencer and merger has been changed from DEBUG to INFO.

Fixes

[FIX]

  • Following a hard reset of the Reliable Transport (RT), the log files in the out folder are truncated rather than deleted. This means that the replicated log files for all subscribers to the RT instance will be garbage collected once the RT archiver starts after the reset. This prevents a hard reset from causing subscriber PVCs to fill up due to old/corrupt log files remaining.

Artifacts

type location
Infrastructure kxi-terraform-1.3.1.tgz
kdb Insights Enterprise insights-1.3.3.tgz
Operator kxi-operator-1.3.1.tgz
CLI kxicli-1.3.1-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0
RT Bridge rt-bridge-1.4.0-install.sh

Batch Ingest Known Issues

  • In the unlikely event of two batch ingests being done within an end of interval (EOI) event (for example, a 10 min window) a one hour delay would occur for the subsequent ingest, but there would not be any data loss, just a delay.
  • If the Storage Manager (SM) crashes during a batch ingest, the ingest will fail. On SM recovery, it will not try again and must be restarted, but no clean-up is required.

Upgrade Notes

  • When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Terraform Script

  • Note the version. GKE cluster cannot be deployed using a 1.3.0 version of the script, a 1.3.* version of the client scripts is needed to use Kubernetes v1.22 on GKE.

Resource Coordinator

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" can be misleading in the timeout message. It should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

Python Scratchpad for Queries

  • If the script consists of multiple statements, the console does not print the output of the last evaluation, and instead it only displays the expression before it is evaluated.

S3 Import Wizard

  • If S3 is selected as the source and next is clicked, no error will be thrown, which is invalid. At least one file needs to be present

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the Insights values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

Users who currently set initUser.enabled=true or initClient.enabled=true within their own values.yaml may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, the user is required to set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here

Assembly blockSize changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

  • if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
  • this has the highest performance, but has no RAM limit on received data
  • if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
  • the smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.
sm:
  size: 1
  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 and kdb Insights Enterprise 1.2.0 if you're upgrading from versions earlier than 1.1.0 or 1.2.0, for notes about