kdb Insights Enterprise - Release Notes

This page details the release notes for the latest kdb Insights Enterprise releases. To find the release notes for an older release, refer to the Previous Releases page.

1.17.0

Release Date 2025-12-12

kdb Insights Enterprise 1.17 delivers greater resiliency for critical services, more storage class options, and a wide variety of improvements across the system. Read on to find out more!

New Features

1. High Availability of Authentication Management
2. Preview option for the Query Window

1. High Availability of Authentication Management

The underlying systems that manage authentication in Insights Enterprise (Keycloak and its PostgreSQL database) have been upgraded to a version that supports high-availability configuration. This delivers greater resiliency for this critical service, and is now the default configuration. This upgrade also makes upgrade and rollback of these components much more seamless.

See Keycloak and PostgreSQL configuration for more details.

Note

Access to the Keycloak administration console has been updated to https://{INSIGHTS_HOSTNAME}/admin.

Important

When rolling back to a version prior to 1.17, the rollback process reuses the existing PersistentVolumeClaim (PVC) from the previous PostgreSQL installation to restore the original database state.

Changes made to the database with the upgraded system will be lost.

2. Preview option for the Query Window

The database preview API is now exposed within the Query window, giving users the ability to view a configurable subset of records based on a date range. This offers a fast and simple method to evaluate a table, view/check data, and identify the columns available. See Preview for details of the API, and Query Window for an overview of the Query window.

Improvements

This release includes a variety of improvements across the entirety of kdb Insights Enterprise.

Expand to see the details of the full set of improvements.

Terraform

Client terraform scripts now support using Lustre-based storage on Azure, AWS and GCP. If selected at install time, the Lustre csi driver will be deployed and a Lustre storage class created. See the Configuration section for GCP, AWS, or Azure depending on your relevant cloud provider.

Stream Processor

• The web interface now supports setting the Kafka Consumer Group ID for Kafka Reader nodes. See the group parameter for the Kafka reader for details.

• autoRegister support has been added for Avro schemas using the Kafka writer. See the autoRegister parameter of the Kafka writer for details.

Database

• When using multiple labels in a query, you can now set the environment variable KXI_ALLOW_PARTIAL_LABELS to allow partial results to be returned in the event that the relevant data for each label does not fully intersect. This will return the data for the label combinations that do exist, and will also provide information on which combinations could not return data. See the KXI_ALLOW_PARTIAL_LABELS parameter for details.

• Added the ability to use date as the groupBy column in getData queries. See the groupBy parameter for getData for details.

• A subset of a schema's tables can now be imported via Initial Import. Any tables that did not yet have data will be created empty at startup, following the database validation operation, allowing for future data ingestion. https://kxl.atlassian.net/browse/KXI-63483

• When defining a User-Defined Analytic, the scope field can now be set as mandatory.

• Added a new helper function kxi.appendLabels for User-Defined Analytics creation, which takes table data and adds the current DAPs assembly labels to the table.

• The database now supports changing a column from scalar to compound.

Package Manager

• Any kxi pm commands that accept a VERSION filter now also accept a requirement specifier on top of an exact version string. See the following examples:

  kxi pm rm my-pkg <2.0.0
  kxi pm rm my-pkg !=1.0.0
  kxi pm pull my-pkg ~=1.0.0
  kxi pm pull my-pkg >0.0.1
  

Scratchpad

• The default for pre-provisioned scratchpad instances has been changed from 3 to 1 to preserve resources.

RT Java SDK

• The ability to pass arguments to a DataAccessClient's ping method has been added, allowing you to ping a particular assembly. See ping for details.

Fixes

This release includes numerous bug, stability and security fixes.

Expand to read details of some important resolved issues here.

Database

• Fixed an issue in the Service Gateway where subscription requests sent to the client might not receive a SUBNAP response.

• There is now protection against accidentally assigning globals in .kxi.qsql. A side-effect of this is that queries such that make use of tableName.column type notation will no longer work. These can be fixed by changing to tableName`column. For example, min trade.price is fixed by switching to min trade`price.

• Fixed an issue where streaming data sources in Views could fail with a ‘Stream closed' error on redeploy of a pipeline.

• Resolved an issue with the KXI_SM_EOI_REQUESTS_PENDING metric, which was incorrectly displaying negative values after emergency EOIs, even though there was no actual processing problem.

• It is no longer possible to batch ingest data where the date of prtnCol differed from the defined partition date. If this is attempted, the batch ingest will be rejected entirely.

• When a schema is changed, the old schema will be honoured for any data added to RT logs before the database restart and schema conversion. Publishers must not begin publishing in the new schema before the database is restarted to perform the schema change; otherwise, a schema mismatch error will occur.

• Improved the error handling of database migration to object storage. Previously, in some cases, if the upload failed, data loss was possible.

• Batch ingest now explicitly rejects incoming data for the time range overlapping with object storage tier. Previously, the Storage Manager discarded such data.

• Added a validation rejecting non-partitioned data provided to batch ingest in a date partition. Previously, such data was erroneously written to HDB partition.

• When EOD writedown discards data older than the configured retention period, SM now prints a warning message.

• Fixed an issue where, with KXI_LATE_DATA disabled, DAP purviews could be incorrect after SM restart, causing query timeouts.

Stream Processor

• Upgraded librdkafka version from 1.9.2 to 2.11.1.

Security

• Security patches have been applied and CVEs have been remediated.

Web Interface

• Fixed an issue in Views where the value for the scope parameter for a UDA does not always get saved and reused.

Important Upgrade and Deployment Considerations

Upgrades

1. Following a Keycloak upgrade, usernames must now be at least 3 characters long. Existing usernames with fewer than 3 characters will remain unchanged.

2. When interacting with the following endpoints using curl:

   • /clientcontroller/leave
   • /clientcontroller/enrol
   • /informationservice/details/{id}

   The request must include the following header: --header "Content-Type: application/json". Requests that do not include this header will fail or be rejected by the service.

3. If your upgrade includes a change to enable or disable encryption, Keycloak may experience issues that manifest as problems in Insights CLI's operation. The potential issues and their suggested resolution are as follows:

   • At Insights CLI Browser Auth Challenge, the challenge URI takes a prolonged time to load in your browser. To resolve, wait 3 minutes then try loading again.
   • At Insights CLI Browser Auth Challenge, the CLI fails with 400 Bad Request. To resolve, run the CLI upgrade operation again.
   • While performing the upgrade, Insights CLI timeouts waiting for Keycloak replicas to become Ready and aborts. To resolve, kill the 3 active Keycloak pods. Kubernetes will start new pods which will resolve the problem.

Default Configurations

The default number of Resource Coordinators (RC) deployed has been changed to 1. This is to align to the default setting of 1 for Data Access Processes (DAPs). In general, there needs to be as many (or more) DAPs as there are RCs. See Ordinal Connection for details. Scratchpads

If kxi install delete or running an upgrade hangs, this may be caused by a hanging scratchpad job which will require manual intervention. This is due to the scratchpad manager’s cleanup job trying to download a bitnami kubectl image that no longer exists. This can be resolved via a manual deletion of the job:

$ kubectl delete job insights-kxi-scratchpad-manager-cleanup-hook -n $MY_NAMESPACE

Following this, clean up both the scratchpads and services manually, which can be done from k9s, or the terminal as follows:

$ kubectl get pods -n $MY_NAMESPACE
NAME                                                              READY   STATUS    RESTARTS   AGE
insights-scratch-user-2d7ebc17a998f9deb79ead9511edbbdd-544r59xc   2/2     Running   0          6m38s

$ kc delete pod insights-scratch-user-2d7ebc17a998f9deb79ead9511edbbdd-544r59xc -n $MY_NAMESPACE
pod "insights-scratch-user-2d7ebc17a998f9deb79ead9511edbbdd-544r59xc" deleted

$ kc get svc -n $MY_NAMESPACE
NAME                                                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE
insights-scratch-user-2d7ebc17a998f9deb79ead9511edbbdd      ClusterIP   10.218.30.120   <none>        80/TCP,8181/TCP   27m

/mnt/c/Users/user [aws-cluster] $ kc delete svc insights-scratch-user-2d7ebc17a998f9deb79ead9511edbbdd -n $MY_NAMESPACE
service "insights-scratch-user-2d7ebc17a998f9deb79ead9511edbbdd" deleted

If you're installing an unpatched version, you'll need to update your values file to correct the image names and disable scratchpad quick startup:

kxi-scratchpad-manager:
  scratchpadQuickStartup:
    enabled: false
  cleanup:
    enabled: true
    image:
      repository: "bitnamilegacy"
      component: "kubectl"
      tag: "1.33.3"
  initDBJob:
    image:
      repository: "docker.io/bitnamilegacy"
      component: "postgresql"
      tag: "15.0.0-debian-11-r1"
      useLocal: true

Database

It is recommended to have an inventory file in place when one of your database tiers resides on object storage to help speed up reload times. Previously, the file information was configured by setting the KX_OBJSTR_INVENTORY_FILE environment variable. This is no longer supported, as both SM and DAPs now handle this by themselves. Refer to Object storage inventory files for more details.

If you have an inventory file in place, it will be necessary to change the configuration of its location reference. In your package file, you will now need to specify the location of the inventory file using the location parameter of the sm.tiers[].inventory property.

This sample configuration illustrates an S3 tier defined in a package file:

- name: s3
    mount: hb
    store: s3://kxi-sm-example/db
    inventory:
     enabled: true
     location: [file location here]

Note

The DAP will no longer start if the environment variable is set directly.

Third-party Dependencies

kdb Insights Enterprise 1.17.0 standalone install supports the following versions of third-party dependencies:

• cert-manager - 1.19.1
• nginx-ingress - 4.14.0
• rook-ceph - 1.18.7
• Kubernetes - 1.33

These versions are used in the 1.17.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

• OpenShift
• OpenSource K8s
• EKS, AKS, and GCP

Artifacts

Type	Nexus location	Downloads Portal location
Enterprise	insights-1.17.0.tgz	insights-1.17.0.tgz
Operator	kxi-operator-1.17.0.tgz	kxi-operator-1.17.0.tgz
CLI	kxicli-1.17.0-py3-none-any.whl	kxicli-1.17.0-py3-none-any.whl
RT q interface	rt.1.17.0.qpk	rt.1.17.0.qpk
RT C interface	kxi-c-sdk 1.17.0	kxi-c-sdk 1.17.0
RT Python interface	kxi-rtpy-1.14.0	kxi-rtpy-1.14.0
RT Java interface	kxi-java-sdk 1.16.0	kxi-java-sdk 1.16.0
RT C# interface	kxi-csharp-sdk 1.17.0	kxi-csharp-sdk 1.17.0
Infrastructure	kxi-terraform-1.17.0.tgz	kxi-terraform-1.17.0.tgz
kxi-management-service	kxi-management-service-1.0.0.tgz	kxi-management-service-1.0.0.tgz
insights-on-k8s	insights-on-k8s-1.1.17.tgz	insights-on-k8s-1.1.17.tgz

Summary

We hope you find some useful features that optimize your kdb Insights Enterprise experience. Try them out and contact our Support Team if you need any help.

We look forward to bringing you even bigger features in kdb Insights Enterprise, coming soon!