kdb Insights Enterprise - Release Notes

This page details the release notes for the latest kdb Insights Enterprise releases. To find the release notes for an older release, refer to the Previous Releases page.

1.14.0

Release Date 2025-06-30

kdb Insights Enterprise 1.14 delivers powerful new language interfaces and improvements (C#, C, Python), embeddable dashboards, enhanced access controls, greater flexibility for data subscriptions, and many performance, usability, and deployment improvements. Read on to find out more!

New Features

1. Embeddable Dashboards
2. New C# Interface
3. Subscription to data streams in C
4. Improved Access Control
5. CLI for Windows
6. Topic Filtering
7. RT Python Interface Enhancements

1. Embeddable Dashboards

Integrate your kdb Insights Enterprise views into any web-based application. This feature enables you to leverage your views further for other systems or user bases, in both internal applications and external-facing portals. Refer to Embedding in an iframe for details.

2. New C# interface

A new C# language interface has been added for publishing & subscribing to data in kdb Insights Enterprise from client applications implemented in C#. This language interface supports cross-platform client application deployment on Windows and Linux. Refer to Using the kdb Insights C# interface for details.

3. Subscription to data streams in C

Client applications written in C using the C language interface can now subscribe to data streams from kdb Insights Enterprise, using the new secure data egress endpoints. Refer to the Subscription section of the Using the C Interface documentation for details. Here is a helpful example of how to establish a subscription in q, or how to configure this using Stream Settings in the web interface.

4. Views-only user role

A new Views-only role has been added, allowing administrators to grant certain users read-only access to Views without access to other entities such as databases or pipelines. This gives administrators better control over access and security. Refer to Roles for details.

5. Windows support for the CLI

The kdb Insights CLI now supports Windows, eliminating the need for WSL. This enhancement reduces complexity and administrative burden, simplifying management and development for kdb Insights Enterprise users working in Windows environments. Read the documentation for instructions on how to install or upgrade the CLI.

6. Subscribe to multiple topics of a data stream

Reliable Transport now allows subscription to multiple topics of the data stream, allowing users to publish/subscribe only to messages of interest. This gives greater control over how data is passed through the system and can boost throughput. This feature is available in the Python, Java, C, and q RT interfaces. Refer to Subscribe for details on how to subscribe to a full stream or particular topics within the stream.

This will be available in the Stream Processor's UI Reader/Writer in a future release.

7. Reliable Transport Python Interface Enhancements

• The RT Python interface can now be used to subscribe to updates from an RT stream and leverage RT topic filtering. Refer to the Subscribe section of the Python API documentation.

• You can now query kdb Insights Enterprise data using the RT Python interface. Refer to the grant_type parameter and the kxi.query.Query API for how to configure an authenticated connection, and the Quickstart guide to help set this up.

Improvements

This release includes a variety of improvements across the entirety of kdb Insights Enterprise!

Expand to read details of the full set of improvements!

Python 3.11 Upgrade

• The Stream Processor and Data Access Processes have been upgraded to run Python 3.11, improving performance and providing access to the latest language features.

Potential Breaking Change

Note that this upgrade may impact any User-Defined Functions/Analytics or Python code in a pipeline that was developed against the prior version. Refer to the Important Upgrade & Deployment Considerations section for full details on potential breaking changes.

Data Flow Observability

• A new API for Reliable Transport has been added to help troubleshoot data flow issues where publisher clients have deduplication enabled. Refer to RT deduplication for details of the API and also to the kdb Insights Reliable Transport documentation for more general information on the RT deduplication capabilities.

SQL2

• .kxi.sql2 queries that contain more than one UNION, EXCEPT, INTERSECT operation are now supported for combining queries. For example:

  (select a, b from table1) union (select a, b from table2) except (select a, b from table3)
  

  Note

  When using EXCEPT ALL in SQL2, the result presently aligns to q/kdb+ behavior (that is, all duplicates are removed) rather than postgresql behavior where duplicates would be included. This will be resolved in a future patch.

• SQL2 queries with multiple set operation clauses are now supported.

  ((w union x) intersect y) except z
  

• .kxi.sql2 now supports current_time in the SELECT and WHERE clause of a query.

Improved Organization for Views

• Views can now be organized into groups using tags. This feature improves the management and arrangement of views, making it easier for users to navigate and locate the views of interest to them. Refer to Tags for details.

Improved performance and efficiency of pipeline ingestion from object storage

• SP object store read performance has been significantly enhanced through the introduction of parallelization in downloading and processing files. This improvement greatly reduces load times for large datasets, enabling faster data ingestion and processing from object storage. To take advantage of these improvements, update your pipelines to use the v2 versions of the Amazon S3, Microsoft Azure Storage or Google Cloud Storage readers. Refer to the guide Ingesting from Object Storage for details.

Configurable Timeout on Test Pipeline

• Pipeline testing now includes configurable timeouts for both Quick and Full tests. Users can adjust timeout settings to accommodate specific environmental circumstances regarding data arrival. Refer to Test Timeout for details.

Query Tabs

• The Query tab now has a Cancel button to interrupt the current query. Note that cancel does not work for Python queries at this time.

• The Query tab now has a Reset button, which restarts the scratchpad process, causing any scratchpad state to be lost. This can be used to reset the scratchpad to a usable state when it is running a query that can't be interrupted with Cancel.

• The default disk size for scratchpads has been increased from 50mb to 250mb to avoid experiencing resource contention when running test deploys or running Python installs that require more space.

• For User-Defined Analytics, [nulls], positive infinity, and negative infinity can now be used for any parameters that have a type where this value is present. They can be entered as, and only as, 0N, 0w, and -0w respectively.

• The Stream Processor's beta features are now available in quick tests and the scratchpad when beta features are enabled. If your values file is set to have betaFeatures: true, then quick tests and the scratchpad honor this.

• Added logging improvements to the scratchpad. Now, all output from STDOUT and STDERR are captured and can be retrieved using .com_kx_edi.getLogs[] and .com_kx_edi.clearLogs[]. This feature is disabled on startup and needs to be started using .com_kx_edi.cacheLogs[].

Database Configuration

• zstd has been added as a database compression option in the web interface.

Packaging

• kxi pm info now allows deeper inspection of a package, namely the ability to look at table and pipeline contents. Refer to PM Info for details.

• kxi pm push CLI command now supports locking Q files during upload. Refer to PM Push and Lock Code for details.

• Entitlements administrators can now create/delete/add/remove entitlements from a group using the kxi entitlement manage command. Row-level entitlements can now also be managed using the kxi entitlement manage command. Refer to Manage for details.

• The Package Manager now supports backup and restore options. Refer to Package Backup and Restore for details.

• It is now possible to add a new pipeline to a running package.

Deployment

• Using the Terraform scripts, it is now possible to deploy into an existing VPC in AWS. Refer to Configure a Kubernetes cluster on AWS for details.

Fixes

This release includes numerous bug, stability and security fixes.

Expand to read details of some important resolved issues here

Stream Processor

• Resolved an issue related to input ordering for nodes that utilize multiple inputs.

Database

• DBM now resumes interrupted data migration processes, preserving progress in cases where the operation was previously halted unexpectedly.

• On the Schema Settings page, Code View now prevents schema changes where a column with type any attempts to set the compound field to true.

SQL2

• Fixed an issue where an error could be experienced in a SQL2 join query where only the table was referenced, and no SELECT … FROM [table name] clause was provided.

• Fixed an issue where a SQL2 query would fail if the column in the WHERE clause did not appear in the SELECT clause.

User-Defined Analytics

• UDAs now support empty lists. Previously, the Run Query button in the web interface would be disabled if there were any empty lists present.

Packaging

• Deleting a package from CLI now explicitly asks for user confirmation. Previously, there was no confirmation before the permanent delete was executed.

Scratchpad

• Basic queries now default to show the last 100 rows to help avoid resource usage issues.

Security

• Various CVEs were remediated as part of this release.

Important Upgrade and Deployment Considerations

Python 3.11 Upgrade

The upgrade to Python 3.11 in the Stream Processor and Data Access Processes brings the following important changes:

1. EmbedPy Removal: Customer code using EmbedPy no longer works.

2. PyKX Upgrade: PyKX has been upgraded from version 2.5.5 to 3.1, which includes breaking changes to PyKX behavior. For more details, refer to this article.

3. Machine Learning API Update:

   The scikit-learn library has been upgraded from 1.5.2 to 1.6.1. This upgrade introduces the following breaking API changes:

   • adaBoostClassifier: The SP operator no longer supports SAMME.R.
   • randomForestClassifier: The maxFeatures parameter now defaults to sqrt instead of auto.
   • Time-Based Features: Time columns are no longer automatically converted to a supported type (integer or float).

getData

The behavior of distinct when used in the agg parameter of the arguments of .kxi.getData is now always returned enlisted. Refer to the following example:

1) With `agg=enlist`distA`distinct`a` would get the following payload returned:

Old Behavior:

distA
-----
1
0
2
4

New Behavior:

distA
-------
1 0 2 4

2) With `agg=((`distA`distinct`a);(`distB`distinct`b))` both old and new behavior would return this:

distA   distB
-----------------
1 0 2 4 4 0 2 1 3

Terraform Scripts

• The bastion host for AWS has been changed from Amazon Linux to Ubuntu.

• If deploying on OpenShift, it is required to define storage class values to ensure they appear in the web interface. Refer to OpenShift Specific Config for details.

Profile Configuration

There is an upgrade requirement for release 1.14 that you must either configure the default profile for the environment you are upgrading, or specify a profile override.

To upgrade with a previously configured default profile, run:

kxi install upgrade --version 1.14.0 --operator-version 1.14.0 --filepath values.yaml

To upgrade with a profile override, run:

kxi --profile <profile name> install upgrade --version 1.14.0 --operator-version 1.14.0 --filepath values.yaml

Configuring a profile

Run the kxi configure --profile <name of new profile> command to configure a new profile. For more information, read Profiles.

Enabled Entitlements before the upgrade of kdb Insights Enterprise

If you are upgrading a system that has Entitlements enabled, or if you are turning Entitlements on as part of the upgrade, ensure you have the Administrator role so that the packages can be redeployed. If you do not have the Administrator role, you will not be able to run and complete the upgrade.

Rollback

As of 1.14, the rollback process has been greatly simplified. To take advantage of this, ensure you are using the 1.14 version of the KXI CLI. Refer to Rollback for details.

Deprecations

• Package manager V1 API is now marked as deprecated for future removal. It is recommended to begin migrating to V2, if possible.

Third-party Dependencies

kdb Insights Enterprise 1.14.0 standalone install supports the following versions of third-party dependencies:

• cert-manager - 1.17.2
• nginx-ingress - 4.11.5
• rook-ceph - 1.17.2
• Kubernetes - 1.32

These versions are used in the 1.14.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

• OpenShift
• OpenSource K8s
• EKS, AKS, and GCP

Artifacts

Type	Nexus location	Downloads Portal location
Enterprise	insights-1.14.0.tgz	insights-1.14.0.tgz
Operator	kxi-operator-1.14.0.tgz	kxi-operator-1.14.0.tgz
CLI	kxicli-1.14.0-py3-none-any.whl	kxicli-1.14.0-py3-none-any.whl
RT q interface	rt.1.14.0.qpk	rt.1.14.0.qpk
RT C interface	kxi-c-sdk 1.14.0	kxi-c-sdk 1.14.0
RT Python interface	kxi-rtpy-1.14.0	kxi-rtpy-1.14.0
RT Java interface	kxi-java-sdk 1.13.0	kxi-java-sdk 1.13.0
RT C# interface	kxi-csharp-sdk 1.14.0	kxi-csharp-sdk 1.14.0
Infrastructure	kxi-terraform-1.14.0.tgz	kxi-terraform-1.14.0.tgz
kxi-management-service	kxi-management-service-1.0.0.tgz	kxi-management-service-1.0.0.tgz
insights-on-k8s	insights-on-k8s-0.4.11.tgz	insights-on-k8s-0.4.11.tgz

Summary

We hope you find some useful features that optimize your kdb Insights Enterprise experience. Try them out and contact our Support Team if you need any help.

We look forward to bringing you even bigger features in kdb Insights Enterprise 1.15, coming soon!