kdb Insights Enterprise - Release Notes

This page details the release notes for the latest kdb Insights Enterprise releases. To find the release notes for an older release, refer to the Previous Releases page.

1.16.1

Release date: 2025-11-10

Improvements

This release includes various improvements, bug and security fixes across the entirety of kdb Insights Enterprise!

Expand to see the details of the full set of improvements.

Stream Processor

• Upgraded the librdkafka library to version 2.11.1.

Database

• Added a new endpoint, /ws/v1/subscriber/nodes, to the Service Gateway. This allows users to query the available websocket streaming endpoints.

Fixes

Expand to view details of key issues that have been resolved.

Database

• Resolved a timing issue with the inter-process signals that could cause processes that were behind in ingesting data to skip ahead in the data stream.

Artifacts

Type	Nexus location	Downloads Portal location
Enterprise	insights-1.16.1.tgz	insights-1.16.1.tgz
Operator	kxi-operator-1.16.0.tgz	kxi-operator-1.16.0.tgz
CLI	kxicli-1.16.1-py3-none-any.whl	kxicli-1.16.1-py3-none-any.whl
RT q interface	rt.1.16.0.qpk	rt.1.16.0.qpk
RT C interface	kxi-c-sdk 1.16.0	kxi-c-sdk 1.16.0
RT Python interface	kxi-rtpy-1.14.0	kxi-rtpy-1.14.0
RT Java interface	kxi-java-sdk 1.16.0	kxi-java-sdk 1.16.0
RT C# interface	kxi-csharp-sdk 1.16.0	kxi-csharp-sdk 1.16.0
Infrastructure	kxi-terraform-1.16.0.tgz	kxi-terraform-1.16.0.tgz
kxi-management-service	kxi-management-service-1.0.0.tgz	kxi-management-service-1.0.0.tgz
insights-on-k8s	insights-on-k8s-0.4.36.tgz	insights-on-k8s-0.4.36.tgz

1.16.0

Release Date 2025-10-29

kdb Insights Enterprise 1.16 delivers performance and security improvements, further extensibility and resiliency, enhancements to APIs, and user workflow improvements. Read on to find out more!

New Features

1. Entitlements no longer in Beta
2. High availability of the Package Manager
3. Improved security for pipelines
4. Configurable timeout on the Query Window

1. Entitlements no longer in Beta

Both data entitlements and package entitlements are now out of beta. Package owners and Entitlements Administrators can configure:

1. Package entitlements: this limits the ability to view, edit, or deploy the package details. The package details include the database, pipeline and view configuration.

2. Database and Row Level entitlements: this limits the ability to query the data a user or pipeline can read from a database.

Refer to the Entitlements documentation for more details.

2. High availability of the Package Manager

The Package Manager is responsible for serving package details to the web interface, as well as for deploying and tearing down packages. The Package Manager can now be configured to be highly available, ensuring that the Web Interface and the CLI are always able to administer packages and provide package details. Refer to the Package Manager reference for more details on how to configure the Package manager to run in this configuration.

3. Improved security for pipelines

This release introduces enhanced security for managing sensitive information in pipeline configurations. You can now reference environment variables in node parameters and have them resolve at runtime. These environment variables can be mounted from Kubernetes secrets created outside the package, avoiding package or pipeline definitions from containing sensitive information.

Refer to secure pipelines with Kubernetes secrets for more information and to environment variable and secrets for details on how to set Kubernetes secrets in the Web Interface.

4. Configurable timeout on the Query Window

In the Query Window you can now specify a custom timeout value ranging from a few seconds to several hours. This timeout value will be used in both Python and q scratchpad queries, as well as database queries. This is particularly useful for long running queries and installing python packages. Refer to execution timeout for more details.

Improvements

This release includes a variety of improvements across the entirety of kdb Insights Enterprise.

Expand to see the details of the full set of improvements.

PDF width improvements in Views

• When saving a View as a PDF you were limited to the standard size of 1920x1080. You can now choose the current preview size, enabling you to save based on much larger screen sizes. Refer to the Views guide for details.

Database

• Performance improvements for table counts: Improved the performance of loading table count caching, for data maintained by Data Access processes.

• The Data Access processes now output a warning or error log each minute if they have not received end of replay message.

-Introduced more robust validation of UDA parameters. A warning is issued if any parameter is missing from registration, and errors if a registered parameter does not exist.

SQL2 Queries

• SQL2 queries now support = when referencing a string column and searching for exact values.

• You can now perform INNER, OUTER, LEFT, RIGHT, CROSS, FULL joins on an arbitrary number of tables, using the = operator.

• There is now support for non-aggregate operations on group by columns.

Refer to SQL2 for details.

Query Window

• In the Basic Query tab, under Optional Parameters, fields that list columns — such as Select Columns — are now searchable. Refer to basic queries for details.

• Placeholders have been added to the Query Window to show you the shortcut keys for executing the current line or selection.

• The default log level for scratchpad deployments is now INFO.

Web Interface

• You can now import a .kxi package into Insights Enterprise from the web interface. See import a package for details.

-The Import Wizard now allows you to change the Node version if multiple versions are available.

-When deleting a package, the user can now choose whether to delete all data related to the package, which includes database contents, pipeline checkpoints, and user states.

Package Manager

• Package roles now control access to the Insights Web Interface. Legacy Package API roles have been deprecated and replaced with updated roles. Users assigned composite roles (e.g., Maintainer) will not be affected by this change.

• You can now use the kxi pm edit command to edit the specifications of a currently running pipeline. This will restart the pipeline in order for the changes to take effect.

• The kxi pm bounce command allows you to target pipelines and databases separately for restarts as well as restart the whole package. You can also restart individual components in a running package using the kxi pm bounce command.

• The kxi pm field info command was added to get up-to-date server side Field information.

• The following changes were made to the kxi package packit command:

  • Target directory can now be specified by --target
  • By default it will keep the current version of your package in the .kxi. Use the checkpoint command if you want automatic version updates
  • The --tag option is now deprecated and is the default behavior
  • The --override-deps/--keep-unlocked/--all-deps options are now deprecated

• The kxi pm journal command and corresponding endpoint shows successful events that have been executed in the Package Manager. This feature requires package admin role.

Refer to the Package Manager Reference page for more details.

Reliable Transport

• RT server logs are now deleted by default.

• We now provide environment variables which, if set by RT clients, will determine the replicator endpoint to connect to. Refer to Using the C interface for more details.

Stream Processor

• The new version of the Parquet reader, v2, now manages the file downloads more efficiently against available disk space across all file readers in the pipeline. The v1 version operates independently from any other readers in the pipeline and could exhaust available space if multiple files are downloaded at the same time. Refer to ingesting data from object storage for more details.

Trraform Scripts

• A new guide is available for helping you upgrade your Kubernetes version for a Client Terraform Script environment. Read more about how to upgrade a Kubernetes cluster andupgrade third-party dependencies.

Fixes

This release includes numerous bug, stability and security fixes.

Expand to read details of some important resolved issues here.

Database

• Switched Service Gateway encoding to UTF-8 to support characters such as 你好.

• Fixed an issue where select *,* from table would fail with a type error if one of the columns was of symbol type.

• Fixed an issue where select exchangeID,* from exchange will now return the exchangeID column twice, matching the Postgres format. The previous behavior would return an exchangeID column and an exchangeID1 column.

• Fixed an issue where UDA parameters using large numbers were being rounded.

• Cleaning up persistent storage now requires insights.package.cleanup role.

• Built-in APIs will now filter out partitioned table records with timestamps of -0Wp instead of hitting a domain error.

Pipelines

• Fetching details of non-existent pipelines wasn't returning a 404 error code, this has been resolved.

Packaging

• Extra fields are now ignored instead of generating errors on Package entities.

Azure

• Fixed issue with redeploy not deploying the database when using an azure cluster.

Security

• Security patches have been applied and CVEs have been remediated.

Web Interface

• Fixed an issue where graphs weren’t being plotted in the Scratchpad after running a database query.

• When running SQL queries in the Query Window, the CTRL+D or ⌘D key will now execute the current line or selection, rather than the entire editor.

• Database configurations can now be exported for running databases.

Upgrade

• When upgrading from an environment where entitlements are disabled to one where entitlements are enabled, the Owner field for existing package entitlements is not automatically populated. After the upgrade a user with the insights.role.administrator role needs to use the entitlements update command to add an entitlements owner to any existing packages.

Important Upgrade and Deployment Considerations

Upgrades

• If upgrading from version 1.14.x or 1.15.x to 1.16, and you have Entitlements enabled, you must disable entitlements first. You can then perform the upgrade to 1.16, and then re-enable entitlements as part of the upgrade.

• To run an upgrade or rollback where entitlements is enabled in either version, when calling kxi install run, you must have the insights.role.administrator role.

• When you upgrade and turn entitlements on, you need to ensure that the user that deploys any pipeline that reads data from a kdb Insights Database Reader has entitlements to read data from the database.

Database

It is recommended to have an inventory file in place when one of your database tiers resides on object storage to help speed up reload times. Previously, the file information was configured by setting the KX_OBJSTR_INVENTORY_FILE environment variable. This is no longer supported, as both SM and DAPs now handle this by themselves. Refer to Object storage inventory files for more details.

If you have an inventory file in place, it will be necessary to change the configuration of its location reference. In your package file, you will now need to specify the location of the inventory file using the location parameter of the sm.tiers[].inventory property.

This sample configuration illustrates an S3 tier defined in a package file:

- name: s3
    mount: hb
    store: s3://kxi-sm-example/db
    inventory:
     enabled: true
     location: [file location here]

Note

The DAP will no longer start if the environment variable is set directly.

Pipelines

Breaking change: For Python pipelines using Kafka writers with a schema registry, the auto_register parameter now defaults to false (previously was true.). This is to match the behavior for Q and UI pipelines where auto register default value is false.

Query failures and Postgres Keycloak

A resilient Postgresql and Keycloak implementation is planned for Insights Enterprise 1.17.0.

For Insights Enterprise 1.16.x, if the Postgresql pod is killed for whatever reason, it is strongly recommended to restart Keycloak as well.

This is to mitigate the possibility of Keycloak getting into a extended stalled state which could impact queries being serviced, and might result in errors in the logs. Typically, Keycloak will recover but this coordinated restart is the best way to protect against this potential failure.

Furthermore, such extended outages can be mitigated by adding the following setting to the values file which reduces the kxi-ent-srv entitlement cache refresh rate from 10s to 60s, reducing the load on Keycloak:

kxi-ent-srv:
    cacheRefresh: 60

Deprecations

Some of the kxi package commands have been deprecated and removed in favor of their kxi pm equivalents.

• kxi package remote-config has been deprecated; use kxi pm config instead

• kxi package remote-convert has been deprecated; use kxi pm convert instead

• kxi package remote-push has been deprecated; use kxi pm push instead

• kxi package remote-pull has been deprecated; use kxi pm pull instead

• kxi package remote-deploy has been deprecated; use kxi pm deploy instead

• kxi package remote-teardown has been deprecated; use kxi pm teardown instead

• kxi package install has been removed

• Changes to kxi package packit:

  • --tag option is now deprecated and is the default behavior.
  • --override-deps/--keep-unlocked/--all-deps options are now deprecated.

Third-party Dependencies

kdb Insights Enterprise 1.16.0 standalone install supports the following versions of third-party dependencies:

• cert-manager - 1.18.2
• nginx-ingress - 4.11.5
• rook-ceph - 1.17.7
• Kubernetes - 1.33

These versions are used in the 1.16.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

• OpenShift
• OpenSource K8s
• EKS, AKS, and GCP

Artifacts

Type	Nexus location	Downloads Portal location
Enterprise	insights-1.16.0.tgz	insights-1.16.0.tgz
Operator	kxi-operator-1.16.0.tgz	kxi-operator-1.16.0.tgz
CLI	kxicli-1.16.0-py3-none-any.whl	kxicli-1.16.0-py3-none-any.whl
RT q interface	rt.1.16.0.qpk	rt.1.16.0.qpk
RT C interface	kxi-c-sdk 1.16.0	kxi-c-sdk 1.16.0
RT Python interface	kxi-rtpy-1.14.0	kxi-rtpy-1.14.0
RT Java interface	kxi-java-sdk 1.16.0	kxi-java-sdk 1.16.0
RT C# interface	kxi-csharp-sdk 1.16.0	kxi-csharp-sdk 1.16.0
Infrastructure	kxi-terraform-1.16.0.tgz	kxi-terraform-1.16.0.tgz
kxi-management-service	kxi-management-service-1.0.0.tgz	kxi-management-service-1.0.0.tgz
insights-on-k8s	insights-on-k8s-0.4.36.tgz	insights-on-k8s-0.4.36.tgz

Summary

We hope you find some useful features that optimize your kdb Insights Enterprise experience. Try them out and contact our Support Team if you need any help.

We look forward to bringing you even bigger features in kdb Insights Enterprise, coming soon!