Skip to content

kdb Insights Enterprise - Release Notes

This page details the release notes for the latest kdb Insights Enterprise releases. To find the release notes for an older release, refer to the Previous Releases page.

1.18.0

Release date: 2026-03-11

kdb Insights Enterprise 1.18 introduces enhancements focused on developer productivity and operational efficiency. This release improves pipeline testing and stream filtering, expands querying and monitoring capabilities, and strengthens reliability, performance, and operational control across the platform.

New Features

KDB-X Workloads (Beta)

KDB-X is a modular analytics platform that allows developers to build real-time, data-intensive, and AI-driven applications using the kdb+ ecosystem. This release introduces support for working with KDB-X modules, enabling you to develop and test source code locally and then deploy it to kdb Insights using the KXI CLI as part of a kdb Insights package. Refer to KDB-X Workloads and this comprehensive development guide to try it out!

Pipeline Testing

To ensure smoother pipeline development and iteration, the various pipeline test workflows have been streamlined into one lightweight, rapid testing workflow. This feature dramatically improves pipeline testing:

  • Combines the speed and interactivity of the old Quick Test but without the limitations.
  • Handles pipelines that mount Kubernetes resources, like secrets, configmaps, volumes.
  • Runs in the user's scratchpad process so it is faster and more interactive. State information can be saved and queried during or after the test.
  • Doesn't need to deploy in Kubernetes, making startup and teardown faster.
  • The test teardown is fully controlled by the user through a web interface button rather than automatically after a certain period.

Note

This replaces the previous Quick Test feature. The Full Test feature has been deprecated as of 1.18.

Folders for Views

Folders have been introduced as a structured form of tagging that provides a familiar, hierarchical way to organize and browse related Views in the left-hand menu.

As the number of Views grows, folders and tags help maintain a scalable structure, reducing clutter and improving navigation. They also simplify access and sharing by allowing users to quickly locate and share relevant Views without sorting through unrelated content.

Stream Topic Filtering

Pipeline developers can now publish or subscribe to specific data streams with greater control. This allows pipelines to get more targeted with their data of interest and reduce the overhead of processing data they do not need. This reduces resources requirements and overall cost of managing all the data that is is being by distributed by RT streams.

Version 2 of the Stream Reader and Writer provide the following additional features:

  • The v2 reader supports server side topic filtering.
  • Allows multiple Stream Readers and Writers in the same pipeline to point to different Streams.
  • Stream-side filtering of messages.
  • In the web interface, v2 nodes allow you to configure the stream directly within the node, rather than through the Settings tab.

Refer to Streams in Pipelines for an example.

Monitoring Stack Installation via the CLI

The kxi monitoring command now provides the option to install the Insights monitoring stack, including Grafana and Prometheus. Refer to the Install guide for more details.

Important

This is not currently supported when using a proxy.

Backup and Restore Improvements

The kxi backup restore-pvc command now supports the restore of the IDB and HDB to a new cluster using a new --reset-rt flag. Refer to the Restore documentation for more details.

Note

If database encryption is turned on and you intend to restore the database to another cluster, you must ensure you have a backup of your encryption key or decrypt the database before you make the backup. Refer to the Backup and Restore Prerequisites for more details.

Improvements

Checkpointing in Data Writedown

  • To improve efficiency and reliability, the database now logs a checkpoint after each table is written to disk. If an End-of-Day (EOD) writedown is interrupted after some tables or dates have been processed, a retry resumes from the last completed checkpoint and preserves the data already written. Previously, the entire EOD writedown would restart from the beginning. This is enabled by default, but can be disabled by setting KXI_SM_EOD_CHECKPOINT to false.

Logging

  • Debugging in the Scratchpad is easier now, with anything logged to STDOUT and STDERR automatically displaying in all open consoles for the user.

Querying

  • When you are targeting a specific package using the scope.assembly argument, you no longer need to append the assembly variable name with -qe setting when the query is targeted to the Query Environment.

  • When calling an API or UDA, the query now automatically routes to the DAPs(s) and Aggregator(s) for which they are registered, without the need to specify the scope.assembly parameter.

  • Moved some Data Access Process initialization logging from DEBUG to TRACE to avoid unnecessary logging.

  • Support for the following SQL Operators has been added:

    • NOW
    • NULLIF
    • COALESCE
    • CASE
    • IS NULL and IS NOT NULL

  • [BETA] A new optional key can be set at the system level, or added to query request options, to enable tracking of the execution time of queries. This tracks and reports:

    • which DAPs are participating in the request, as well as the RC/AC codes and the duration of the execution.
    • which Aggregator is responsible for aggregating, as well as the duration of aggregation.
    • which RCs are participating in the request (primary and secondaries) including processing and queuing time.

    Refer to Timing UDA Functions for an example using a user-defined analytic for demonstration.

  • An option has been added to API calls to be able to choose the aggregator that the query is directed to. Refer to Request Header for details.

Note

Use caution with this feature and clearly understand your data model and storage configuration. This is most applicable to development or debugging activity. For example, where you may want to intentionally work with a given AGG where you have loaded code or debug/break statements etc.

Web Interface

  • The Preview tab is now the selected tab when a Query window is created. The Preview tab is a lightweight query option to fetch small samples of a table quickly using minimal resources. For more advanced options, you can switch to the Basic tab.

  • The Basic tab has been enhanced to allow selection of a time range and to limit the number of rows returned per request. The default query now retrieves data from the last 24 hours, limited to 100 rows.

  • Added the ability to duplicate CLI-created pipelines within the web interface and add them to new or existing packages. Refer to Copying Pipelines into Packages for details.

  • As part of the Version 2 Stream Reader and Writer you can now select the Stream in the Node properties and read to and write from more than one stream in a single pipeline.

  • Improved the performance of the Database Settings page when there is a large number of tables.

  • Support for the structuredText format is now available in the Preview tab of the Pipeline editor and the Results tab of the Query window, ensuring types are represented correctly in these tabs.

  • When defining a Database in the web interface you can now choose whether the Streams are configured with 1-node or 3-node. This allows you to reduce the resources a database uses if you do not require high availability for your Streams. For example, in a deployment used for development. Refer to Cluster size for more details.

  • Pipeline Settings validation improvements:

    • Validation was added for the Worker scaling values to ensure it is set to an integer.
    • Minimum and Maximum fields need to be validated together to ensure the minimum value is not larger than the maximum value.

Scratchpad

  • Scratchpads now include audit log messages, which include user IDs and correlation IDs.

  • The Scratchpad now provides more clarity through notifications when cancelling a pending q or Python query in the Query Window.

Packaging

  • The package init command now supports initializing a package from a non-empty directory. Refer to Creating a new package for details.

  • kxi pm list can now filter on package owner.

Database

  • Added new metrics to the Resource Coordinator, Aggregator and DAPs to track application and return code errors. These metrics count the number of API failures. You can configure which application and return codes to track. Refer to API metrics for more details.

Stream Processor

  • The MQTT reader is not compatible with stream or database writers that have deduplication enabled. This restriction prevents potential data loss from MQTT during recovery scenarios. Previously, a validation issue in the web interface allowed this unsupported configuration. This issue has been fixed, and such pipelines will no longer run. If you need to run this type of pipeline and accept the associated data loss risks, disable deduplication on the writer by setting the KXI_ALLOW_NONDETERMINISM environment variable.

Monitoring

  • The HighCPUThrottling alert has been modified to fire if CPU Throttling occurs above 10% for 5 minutes.

Terraform Scripts

  • The Kubernetes default version has been updated to 1.34.

Install/Upgrade

  • kxi install is no longer permitted if the target version is greater than the CLI version.

  • If the backup of packages fails during a rollback, you can still choose to carry on with the rollback.

  • When performing an uninstall, the kdb Insights Enterprise URL and namespace are now provided in the confirmation message to protect against any unintentional operations impacting other instances.

Reliable Transport

  • Caching changes have been made to help reduce disk I/O usage by Reliable Transport. RT can now cache message headers and optionally cache the message payload. Refer to Sub-server Caching for details.

  • Resiliency improvements have been made for power failure scenarios that can cause corruption of RT log files. This reduces the possibility of data flow interruption if there is a loss of power to the RT client and server nodes.

Fixes

  • RT

    • When the configured maximum percentage threshold for archival is reached, RT logs are now truncated correctly.

  • Scratchpad

    • Fixed an issue where the configured timeout value was not preserved after a scratchpad reset.

    • The .kxi.packages.* commands in the scratchpad now execute correctly.

  • Web Interface

    • Fixed an issue where ChartGL charts were not rendered in generated PDF files.

    • The Labels dropdown in the Query Basic tab now displays all labels available on all deployed databases.

    • Switching between different UDAs no longer causes the Apply button to be disabled.

  • Package Manager

    • For Windows systems, defining UDFs in subpaths (for example, my-package/src/my-udf.q) no longer errors with a file not found error upon package deployment.

    • Fixed an issue where defining a single-line UDF caused an error.

  • Querying

    • Fixed an issue where if the first line of qSQL was a comment, the query failed.

    • All UDA queries now properly honor the timeout parameter.

    • If DAPs or Aggregators are not connected to a Resource Connector, a warning is now displayed.

  • Database

    • Data migration between HDB tiers with different compression settings no longer fails to apply a sorted attribute on a symbol column.

    • Batch ingest now correctly recognizes new symbols when executed immediately after an EOI.

    • Fixed an issue during a schema conversion where validation was skipped when multiple conversions were combined, causing the conversion to fail.

    • Fixed an issue where execution of queries was prevented after certain malformed queries were passed to .kxi.selectTable.

    • Terminating a Storage Manager process no longer results in unnecessary errors being logged by the DAPs.

    • Fixed an issue where the timeout for downloading of packages by the Storage Manager and DAPs was set to 30 seconds and not the configurable timeout value.

    • Fixed an issue where when hdr.cast was set to true, distinguished arguments (table,inputTZ,scope, etc.) would not get cast.

    • Fixed an issue where the Storage Manager (SM) did not reject schemas containing a column whose oldName property matched an existing column name. The oldName property must now be removed if it remains from a previous column rename.

Important Upgrade and Deployment Considerations

PortgresSQL DBs

kdb Insights Enterprise uses a CloudNativePG operator to manage the redundant PostgreSQL databases that are in charge of Keycloak authentication data. This uses a write-ahead log file (WAL) to track transactions for recovery purposes. In 1.18, the WAL retention size now defaults to 1GB to prevent unbounded growth. In earlier releases, the WAL could grow indefinitely and consume excessive disk space.

When you upgrade to 1.18, the new default retention limit is applied. However, the limit cannot be enforced automatically if the WAL has already exceeded 1 GB. In that case, you have two options:

  1. If the system is not running (no Keycloak activity), you can delete some or all of the WAL file log entries to free up space.
  2. If the system is running, it is not safe to delete from the WAL file. In this case, provision additional disk space for the WAL directory until you can safely reduce the file size below the default limit.

CNPG Pod Failures

The CloudNativePG pods are deployed in a redundant cluster. If any one of the CPNG pods are failing, it is safe to restart the pod as it will recover state from one of the other pods in the cluster. In the unlikely event of a full pod cluster failure, refer to the recovery documentation.

Deprecations

  • The Full Test feature for pipelines has been deprecated in 1.18. Quick Test has been rolled into the new and improved pipeline testing feature.

  • KXI_SM_INGEST_CLEANUP_AFTER is now deprecated and the retention period is based on KXI_SM_BATCHUPD_CLEANUP_AFTER instead.

  • Removed the deprecated .qsp.read.fromRT and .qsp.write.toRT operators.

Third-party Dependencies

kdb Insights Enterprise 1.18.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.18.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

Artifacts

Type Nexus location Downloads Portal location
Enterprise insights-1.18.0.tgz insights-1.18.0.tgz
Operator kxi-operator-1.18.0.tgz kxi-operator-1.18.0.tgz
CLI kxicli-1.18.0-py3-none-any.whl kxicli-1.18.0-py3-none-any.whl
RT q interface rt.1.18.0.qpk rt.1.18.0.qpk
RT C interface kxi-c-sdk 1.18.0 kxi-c-sdk 1.18.0
RT Python interface kxi-rtpy-1.14.0 kxi-rtpy-1.14.0
RT Java interface kxi-java-sdk 1.16.0 kxi-java-sdk 1.16.0
RT C# interface kxi-csharp-sdk 1.18.0 kxi-csharp-sdk 1.18.0
Infrastructure kxi-terraform-1.18.0.tgz kxi-terraform-1.18.0.tgz
kxi-management-service kxi-management-service-1.0.0.tgz kxi-management-service-1.0.0.tgz
insights-on-k8s insights-on-k8s-1.1.28.tgz insights-on-k8s-1.1.28.tgz

We look forward to bringing you even bigger features in kdb Insights Enterprise, coming soon!