kdb Insights SDK - Release Notes
This page details the release notes for the latest series of kdb Insights SDK releases. To find the release notes for an older release, refer to the previous releases page.
1.16.0
Release Date 2025-10-29
kdb Insights SDK 1.16 delivers performance and security improvements, further extensibility, and enhancements to APIs. Read on to find out more!
New Features
1. Improved Security for SP Pipelines
This release introduces enhanced security for managing sensitive information in pipeline configurations. You can now reference environment variables in node parameters and have them resolve at runtime. These environment variables can be mounted from Kubernetes secrets created outside the package, avoiding package or pipeline definitions from containing sensitive information.
Refer to using environment variable and secrets for details on how to set this for your pipeline.
Improvements
This release includes a variety of improvements across the entirety of kdb Insights SDK!
Expand to see detail of the full set of improvements!
Database
-
Performance improvements for table counts: Improved the performance of loading table count caching, for data maintained by Data Access processes.
-
The Data Access processes now output a warning or error log each minute if they have not received end of replay message.
-
Introduced more robust validation of UDA parameters. A warning is issued if any parameter is missing from registration, and errors if a registered parameter does not exist.
SQL2 Queries
-
SQL2 queries now support = when referencing a string column and searching for exact values.
-
You can now perform INNER, OUTER, LEFT, RIGHT, CROSS, FULL joins on an arbitrary number of tables, using the = operator.
-
There is now support for non-aggregate operations on group by columns.
Refer to SQL2 for details.
Reliable Transport
-
RT server logs are now deleted by default.
-
We now provide environment variables which, if set by RT clients, will determine the replicator endpoint to connect to. Refer to Using the C interface for more details.
Stream Processor
- A new v2 version of the Parquet reader now manages the file downloads more efficiently against available disk space across all file readers in the pipeline. The v1 version operates independently from any other readers in the pipeline and could exhaust available space if multiple files are downloaded at the same time. Refer to ingesting data from object storage for more details.
Fixes
This release includes numerous bug, stability and security fixes.
Expand to see detail of some important resolved issues here
Database
-
Switched Service Gateway encoding to UTF-8 to support characters such as 你好.
-
Fixed an issue where select , from table would fail with a type error if one of the columns was of symbol type.
-
Fixed an issue where select exchangeID,* from exchange will now return the exchangeID column twice, matching postgres format. The old behavior would return an exchangeID column and an exchangeID1 column.
-
Fixed an issue where UDA parameters using large numbers were being rounded.
-
Cleaning up persistent storage now requires insights.package.cleanup role.
-
Built-in APIs will now filter out partitioned table records with timestamps of -0Wp instead of hitting a domain error.
Pipelines
- Fetching details of non-existant pipelines wasn't returning a 404 error code. This has been resolved.
Security
- Various CVEs were remediated as part of this release.
Important Upgrade and Deployment Considerations
Database
It is recommended to have an inventory file in place when one of your database tiers resides on object storage to help speed up reload times. Previously, the file information was configured by setting the KX_OBJSTR_INVENTORY_FILE environment variable. This is no longer supported, as both SM and DAPs now handle this by themselves. Refer to Object storage inventory files for more details.
If you have an inventory file in place, it will be necessary to change the configuration of its location reference. In your package file, you will now need to specify the location of the inventory file using the location parameter of the sm.tiers[].inventory property.
This sample configuration illustrates an S3 tier defined in a package file:
name: s3
mount: hb
store: s3://kxi-sm-example/db
inventory:
enabled: true
location: [file location here]
Piplines
Breaking change: For Python pipelines using Kafka writers with a schema registry, the auto_register parameter now defaults to false (previously was true.). This is to match the behavior for Q and UI pipelines where auto register default value is false.
Artifacts
KX Nexus
The KX Downloads Portal is the preferred repository for all use cases, and KX Nexus links will be removed once the KX Nexus repository is decommissioned. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users.
Stream Processor
| service | Nexus Docker | Downloads Portal Docker |
|---|---|---|
| Controller | registry.dl.kx.com/kxi-sp-controller:1.16.0 | portal.dl.kx.com/kxi-sp-controller:1.16.0 |
| Worker | registry.dl.kx.com/kxi-sp-worker:1.16.0 | portal.dl.kx.com/kxi-sp-worker:1.16.0 |
| Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.16.0 | portal.dl.kx.com/kxi-sp-coordinator:1.16.0 |
| Worker-Python | registry.dl.kx.com/kxi-sp-python:1.16.0 | portal.dl.kx.com/kxi-sp-python:1.16.0 |
| Worker-ML | registry.dl.kx.com/kxi-ml:1.16.0 | portal.dl.kx.com/kxi-ml:1.16.0 |
Database
| service | Nexus Docker | Downloads Portal Docker | Helm Chart |
|---|---|---|---|
| Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.16.0 | portal.dl.kx.com/kxi-da:1.16.0 | |
| Single container | registry.dl.kx.com/kxi-da-single:1.16.0 | portal.dl.kx.com/kxi-da-single:1.16.0 | |
| Storage Manager | registry.dl.kx.com/kxi-sm-single:1.16.0 | portal.dl.kx.com/kxi-sm-single:1.16.0 | |
| Gateway | registry.dl.kx.com/kxi-sg-gw:1.16.0 | portal.dl.kx.com/kxi-sg-gw:1.16.0 | |
| Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.16.0 | portal.dl.kx.com/kxi-sg-rc:1.16.0 | |
| Aggregator | registry.dl.kx.com/kxi-sg-agg:1.16.0 | portal.dl.kx.com/kxi-sg-agg:1.16.0 | |
| Rollback | registry.dl.kx.com/kxi-db-rollback:1.16.0 | portal.dl.kx.com/kxi-db-rollback:1.16.0 | |
| Reference Deployments | N/A | Docker Bundle | N/A |
| Reference Deployments | N/A | N/A | Kubernetes Bundle |
Reliable Transport
| service | Nexus Docker | Downloads Portal Docker | Helm Chart |
|---|---|---|---|
| Reliable Transport | registry.dl.kx.com/kxi-rt:1.16.0 | portal.dl.kx.com/kxi-rt:1.16.0 | kxi-rt-1.16.0.tgz |
Miscellaneous
| service | Nexus Docker | package |
|---|---|---|
| Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.16.0 | |
| OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |