kdb Insights Enterprise - Release Notes

This page details the release notes for the latest kdb Insights Enterprise release. To find the release notes for an older release, refer to the previous releases page.

1.13.1

Release Date 2025-04-28

Improvements

Expand to see detail of the full set of improvements!

Reliable Transport

You can now subscribe to a list of tables in the RT stream. Previously, topic filtering supported a subscription only to a single table.

Database

zstd has been added as a database compression option in the web interface.

Fixes

This release includes numerous bug, stability, and security fixes.

Expand to see detail of some important resolved issues here

Database

Resolved an issue where the selectTable API could return an incorrect result if executing an aggregation that included data on disk. For example, a call such as .kxi.selectTabletableagg!(trade;enlist[x]!enlist(avg;price)) could be subject to this defect.

Packaging

In the CLI, users are now prompted to confirm they intend to delete a package before the deletion operation is executed.

Web Interface

The Database page no longer displays errors when you're opening a database created from the CLI.
When copying a pipeline from one package to another, the new pipeline was not visible in the Associated Pipelines list until a page refresh was done. This has been resolved.
On the Schema Settings page, the Code View for databases now prevents applying a schema change if a column with type Any attempts to set the compound field to true.
When deploying a new database, the Recently Deployed table on the landing page was briefly displaying the database with a status of READY even though the database was not fully deployed yet. This has been resolved.

Third-party Dependencies

kdb Insights Enterprise 1.13.1 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.13.1 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

Artifacts

Type	Nexus location	Downloads Portal location
Enterprise	insights-1.13.1.tgz	insights-1.13.1.tgz
Operator	kxi-operator-1.13.1.tgz	kxi-operator-1.13.1.tgz
CLI	kxicli-1.13.1-py3-none-any.whl	kxicli-1.13.1-py3-none-any.whl
RT q interface	rt.1.13.0.qpk	rt.1.13.0.qpk
RT C interface	kxi-c-sdk 1.13.0	kxi-c-sdk 1.13.0
RT Python interface	kxi-rtpy-1.12.0	kxi-rtpy-1.12.0
RT Java interface	kxi-java-sdk 1.13.0	kxi-java-sdk 1.13.0
Infrastructure	kxi-terraform-1.13.1.tgz	kxi-terraform-1.13.1.tgz
kxi-management-service	kxi-management-service-0.2.8.tgz	kxi-management-service-0.2.8.tgz
insights-on-k8s	insights-on-k8s-0.3.5.tgz	insights-on-k8s-0.3.5.tgz

1.13.0

Release Date 2025-04-08

kdb Insights Enterprise 1.13 gives you tighter control over system processes, more flexible deployment and configuration, and a host of impactful upgrades designed to boost performance, ease-of-use and efficiency. Read on to find out more!

New Features

Native Kubernetes Support
Manually Trigger EOD Processing
Views: Permissioned Exporting of Views Data
Backup/Restore in the CLI
Event-based Triggers for Pipelines

1. Native Kubernetes Support

Support for Native Kubernetes distribution has been added, eliminating the dependency on commercial K8s capabilities. This change provides greater flexibility in determining the right Kubernetes runtime for deployment. Refer to Kubernetes Infrastructure Prerequisites and Installation Validation for details.

2. Manually Trigger EOD Processing

You can now manually trigger Storage Manager to perform end of day (EOD) writedown. This functionality enables requesting an emergency EOD action to push data to the HDB on-disk before the scheduled time, thereby removing memory pressure. This is particularly useful when dealing with large amounts of late data that would be held in the HDB's memory. Read Manual EOD Trigger for details.

3. Views: Permissioned Exporting of Views Data

Views now support role-based file export functionality. This restricts users from exporting data they do not have access to. Read Guide to building Views for details.

4. Backup/Restore in the CLI

The KXI CLI backup and restore now supports backup and restore to all three cloud providers through command-line operations.

5. Event-based Triggers for Pipelines

Pipelines now allow you to auto-trigger execution based on a pre-defined event like EOD or batch ingest completion. Refer to the Pipeline Triggering documentation for details.

Improvements

This release includes a variety of improvements across the entirety of kdb Insights Enterprise!

Expand to see detail of the full set of improvements!

System Resources & Query Environments

Query Environments (QEs) are now disabled by default on a new install to optimize resource usage and improve overall system efficiency. Read the Query Environments documentation for details on how to configure your deployment to include/exclude QEs.

Sample Packages

Sample packages are now available on the KX Downloads Portal to aid in setting up the ingestion examples for both Kafka and S3-based data. Refer to Batch S3 ingest and Streaming Kafka Ingest for guidance on these examples and where to locate the reference packages. Don't forgot about these other pre-existing quickstarts/packages for publishing & querying with the Python API and streaming data from kdb Insights Enterprise over WebSockets too!

CLI Support for Accessing Logs

The KX CLI has been enhanced with a new command for easy log retrieval. This allows developers and admins to easily access and view logs, improving the observability of deployed packages and pipelines. Read Observability Logs for details.

Data Services Events

Data service events allow you to customize and automate the behavior of the system, and trigger other system operations off of them. You can add customized code to run at key points in the data processing lifecycle (like during initialization). This means you can tailor the system's responses, making your data workflows more flexible and dynamic.

Delete Rows in a Table

The Delete Rows in a Table API has been improved now allowing permissioned users to delete rows from tables in the IDB and HDB (deletion from RDB and object storage is not supported). This API remains a beta feature. See Delete Rows in a Table for details.

New Web Interface Landing Page for Improved Usability

A new overview landing page has been introduced to improve the experience for both first-time and experienced users.

Entitlements

Entitlement owners can now manage the entitlements they own without the requiring the insights.entitlement.admin role. See Entitlements for details on Entitlements, and Roles for the specific application roles and their capabilities.

Tier Support in Scope

The scope.tier parameter has been introduced to allow you to target one specific tier in an API call. See Scope for details.

Query Improvements

You can now use SQL value functions current_date, current_timestamp, and current_time so queries of the form select * from trade where date=current_date now works in SQL2. See Value Functions for details.
qSQL queries now enforce database entitlements, enhancing security measures.
There is now a distributed option in the scratchpad for qsql queries, enabling the query to run against the RDB, IDB and HDB. Read the details of the args.scope.tier parameter in the qSQL documentation. Note that responses from multiple DAPs are removed.

Reliable Transport Enhancements

Improvements have been made to Reliable Transport to help reduce latency in message flow, especially when dealing with hundreds of publishers.
If the RT archiver settings are left blank, they now adopt the defaults introduced in 1.13. The retention period is 7 days and the max percentage of available disk is 90%. This protects against overly-rapid truncation of RT logs. See Retention Duration for details on this parameter.

Streaming to Views Enhancements

Support for viewstates as part of the filtering capability of Streaming data sources. This improvement allows dashboard designers to utilize viewstate variables and filter more easily. Read the Filtering based on a drop-down section of the Streaming subway data guided walkthrough.
Caching support in Streaming to Views now allows you to see the recent historical view of the data represented together with updates, giving users a more meaningful view of how data is changing over time. Read the Backtest trading strategies tutorial for an example of adding a View that includes streaming data with historical data included.
Users can now modify the streaming cache on the Subscriber - read the Updating the Cache section. This functionality enables the deletion or replacement of all or specific rows from the cache. This is particularly useful for managing expired futures contracts or performing backtesting.

Scratchpad Enhancements

In q editors, you can now select a q keyword or operator and press Alt + / to open the doc for the selection.

Packaging Enhancements

A new command kxi pm info shows detailed information about a package on the server.
Maintaining or tearing down individual pipelines or databases of a package deployment without impacting other entities is now supported. Read Tear down a specific pipeline for details.

User-Defined Analytics (UDAs) Enhancements

Python pipelines can now read from databases using UDAs. See Readers for details.
The Query tab now supports additional parameter types, as well as parameters that can take on multiple types. For full details, read Supported q types.
All of the distinguished parameters now exist as optional parameters on the Query tab and can be used to customize the routing for every available query.
Support for calling UDAs has been improved with various web interface updates in the query window. These include:
- Encapsulated panels with resizing handles
- An expand/collapse toggle in the Output panel
- The ability to save user-configured workspaces

Terraform Deployment Script Updates

Terraform scripts for AWS, Azure, and GCP have been enhanced with architectural profiles representing the three most common confirmation patterns: High Availability, Performance, and Cost-Optimized. Refer to the Architecture Profiles documentation for details.

Fixes

This release includes numerous bug, stability and security fixes.

Expand to see detail of some important resolved issues here

Insights Core

Insights 1.13 has been updated to run Insights Core 4.1.10, which contains fixes related to the object storage library. Refer to the Insights Core release notes for details of these fixes.

Database

.kxi.getData would hit a length error when trying to apply more than one filter to a reference table joined using a foreign key.
Schema change performance has been improved to support multi-threaded operations, and to only execute against only the impacted tables.
The database now waits to initialize connection to Reliable Transport until its processes are fully online. Previously, DAPs would sometimes try to connect before fully available, resulting in an excessive number of RT logs being generated and issues with their processing.

Reliable Transport

When subscribing to the latest position in RT, all log files, rather than just the log files with the latest data, were copied to the subscriber node. Now, only the log files required are copied.
Resolved an issue in which an incorrect RT_LOG_PATH was set after a restart by qsup.

Web Interface

On the Schema Settings page, it was not possible to sort on multiple columns when any column had the parted attribute set. This restriction has been removed, allowing for sorting on multiple columns regardless of attribute.
On the Pipelines page, there was previously no indication when a pipeline only partially succeeded in its operations. Now, pipelines report partial status, making confirmation or debugging clearer.

Security

CVE-2024-38541, CVE-2024-38541, and various other CVEs were remediated as part of this release.

Upgrade & Deployment Changes

Terraform Scripts

This release removes support for rook-ceph using local SSDs for all three Architecture Profiles. Instead of using local SSD rook-ceph, all rook-ceph deployments will be backed by managed disk to protect against the potential of data loss. Also, the rook-ceph MDS cache size has been set to 4GB to improve stability.

If you are running an environment using local SSD rook-ceph, you should not use this version of the Terraform scripts to upgrade.

Profile Configuration

There is a new upgrade requirement for release 1.13 that you must either configure the default profile for the environment you are upgrading, or specify a profile override.

To upgrade with a previously configured default profile, run:

kxi install upgrade --version 1.13.0 --operator-version 1.13.0 --filepath values.yaml

To upgrade with a profile override, run:

kxi --profile <profile name> install upgrade --version 1.13.0 --operator-version 1.13.0 --filepath values.yaml

Configuring a profile

Run the kxi configure --profile <name of new profile> command to configure a new profile. For more information, read Profiles.

Rollback

If performing a rollback from 1.13.x to any earlier version, assemblies must not be running prior to initiating the rollback to ensure no data is lost. Refer to Upgrading for how to perform upgrades/rollbacks and the available commands to use.

Deprecations

The delete API for performing data deletion from the IDB and HDB will be renamed to deleteRows as of Insights 1.14. This is an advanced warning of the name change.

Third-party Dependencies

kdb Insights Enterprise 1.13.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.13.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

Artifacts

Type	Nexus location	Downloads Portal location
Enterprise	insights-1.13.0.tgz	insights-1.13.0.tgz
Operator	kxi-operator-1.13.0.tgz	kxi-operator-1.13.0.tgz
CLI	kxicli-1.13.0-py3-none-any.whl	kxicli-1.13.0-py3-none-any.whl
RT q interface	rt.1.13.0.qpk	rt.1.13.0.qpk
RT C interface	kxi-c-sdk 1.13.0	kxi-c-sdk 1.13.0
RT Python interface	kxi-rtpy-1.12.0	kxi-rtpy-1.12.0
RT Java interface	kxi-java-sdk 1.13.0	kxi-java-sdk 1.13.0
Infrastructure	kxi-terraform-1.13.0.tgz	kxi-terraform-1.13.0.tgz
kxi-management-service	kxi-management-service-0.2.8.tgz	kxi-management-service-0.2.8.tgz
insights-on-k8s	insights-on-k8s-0.3.5.tgz	insights-on-k8s-0.3.5.tgz

Summary

We hope you find some useful features that optimize your kdb Insights Enterprise experience. Try them out and email our Support Team if you need any help.

We look forward to bring you even bigger features in kdb Insights Enterprise 1.14 coming soon!