Skip to content

kdb Insights Enterprise - Release Notes

This page details the release notes for the latest kdb Insights Enterprise releases. To find the release notes for an older release, refer to the Previous Releases page.

1.20.0

Release date: 2026-09-21

kdb Insights Enterprise 1.20 focuses on deployment flexibility, query scalability and pipeline manageability. It introduces templated packages and read replicas, independent memory profiles for Query Environments, a natural language interface via MCP, granular status tracking for long-running pipelines, and stack traces on failed API queries.

Breaking Changes and Upgrade Notes

Reliable Transport Publisher Logs

RT publishers no longer need to retain their local client logs between restarts. Because publisher state is now tracked differently, you cannot move directly between 1.18 and 1.20 in either direction: both upgrades and rollbacks must be staged through 1.19.

Package Manager Shared Storage Removed

The sharedfiles persistent volume has been removed from the Package Manager chart, and disk-backed serving of packages is no longer supported. Deployments on 1.15 must upgrade through an intermediate release; a direct 1.15 to 1.20 upgrade or rollback is not supported.

Insights Nexus Download Portal Retired

The nexus.dl.kx.com download site has been decommissioned. All Helm charts, images, wheels and raw artifacts are served from the KX Downloads Portal. Update any install scripts, CI pipelines or private mirrors still pointing at the Nexus host.

Client Terraform Scripts now run OpenTofu

The client Terraform scripts now invoke the OpenTofu binary in place of Terraform. Install OpenTofu on any host or runner that runs the scripts. Existing state is read by tofu rather than terraform.

Install/Upgrade

Helm SDK and CLI version 4 are now supported for new installations of kdb Insights Enterprise.

New Features

  1. Natural Language Interface via MCP - BETA

    The kdb Insights Enterprise REST API is now exposed through an MCP (Model Context Protocol) server. You can query pipeline state, diagnose issues and manage publishers in plain language, without constructing API calls by hand.

    The REST API itself is unchanged and remains fully supported. Refer to MCP for more details.

  2. Templated Packages

    A package can now be defined once and deployed in variations, such as a different schema per environment or a different replica count per region, without maintaining overlay patch files or near-identical copies of the same package.

    Templates are rendered with values you supply from the CLI, either once when you push the package or per deployment, so a single definition covers every variant it needs to serve.

    Refer to package templates for more details.

  3. Read Replicas for Targeted Queries and Workloads

    You can now deploy a read replica: an additional set of Data Access Processes (DAPs) that read the same real-time, interval and historical data as an existing deployment, without changing the primary deployment and without copying the data. Use a replica to service long-running or expensive ad-hoc queries, and target those workloads at it rather than overloading the primary system.

    Replicas are disposable and can be sized for the workload they serve, so heavy concurrent queries can be spread across several replicas. Route a query to a replica by naming its assembly in the scope parameter, then tear the replica down when the work is done.

    Read replicas are delivered as a bundled package template. Refer to the read replicas walkthrough for more details.

  4. Independent Memory Profiles for Query Environments

    Query Environment (QE) DAPs can now be sized independently of the main DAPs. Previously, a single query that needed a large amount of RAM forced every DAP, main and QE alike, to be provisioned at that level.

    You can now set separate resource requests and limits for QE DAPs, reserving a large memory profile for the processes that run heavy queries while the main DAPs stay right-sized. This avoids out-of-memory failures on large queries without inflating the footprint of the whole deployment.

    Set resource requests and limits for Query Environment Replicas in Query Settings in the web interface, or in the query environments section of your database configuration.

    Note

    QE settings are applied only when Query Environments are enabled and the replica count is greater than zero. With QE disabled, or a replica count of zero, no QE resources are requested.

  5. Granular Pipeline Status Tracking

    Pipelines now report status for each individual unit, and branches within a pipeline can be triggered independently. This means work that previously needed separate pipelines can be consolidated into one long-running pipeline and sequenced internally.

    Consolidating pipelines this way reduces the number of persistent volumes a deployment needs, because each pipeline previously required at least two. It also removes the startup, shutdown and disk attach/detach latency of a start/stop scheduling model.

    Refer to Pipeline Units for more details.

  6. Detailed Status for Batch Ingestion Pipelines

    The Batch Ingest diagnostics table in the web interface now shows the detail needed to identify individual batch runs:

    • Start and end time for each run, to sub-second precision.
    • Session ID, displayed beneath the pipeline name.

    Refer to Batch Ingest Diagnostics for more details.

  7. Stack Traces on API Query Failures

    A failed API query now returns a backtrace identifying the line that raised the error, rather than only the error type. Previously, a UDA that raised a type error reported Unexpected error (type) encountered executing .kxi.qsql with no indication of where the failure occurred.

    Backtraces are returned on the query API path in both the web interface and the VS Code extension, matching the behavior the Scratchpad already provided for plain q. The .Q.trp and .Q.trpd workaround is no longer required.

    Note

    Backtraces require the insights.query.backtrace role, which is included in the pre-configured Developer role and above.

  8. RT Publisher Resilience

    An RT publisher that has lost or deleted its local log directory now reconnects and resumes publishing from where it left off. Previously, restarting a publisher whose RT logs had been removed caused RT to reject the connection, and no new data reached the stream.

    Note

    This changes the supported upgrade path. See Breaking Changes and Upgrade Notes at the top of this release for more details.

  9. RT Observability

    A new API, /rt-clients-meta, returns sequencer state information from Reliable Transport, including the state held for a given publisher. This gives you a supported way to inspect stream position and publisher registration when inspecting your data ingestion stream.

Improvements

kdb Insights Enterprise 1.20 brings improvements across the product. Database work focuses on observability, with new database migration metrics, identification of the Gateway, Resource Coordinator and Aggregator that served a query, and an opt-in low-memory sort optimization. The Pipeline Editor gains additional Database Reader parameters and support for pipeline annotations, the CLI adds group and role management, and the Stream Processor adds latency profiling. Expand the following sections for more details.

Database

  • A new Storage Manager metric, kxi_sm_dbm_status, exposes the status of DBM tier migrations. The gauge reports statuses of idle, active, or paused when a migration has been suspended due to an EOD or a DBM process restart. A paused migration resumes automatically once the interrupting process completes.

  • Three additional DBM metrics improve visibility into DBM health and backlog: kxi_sm_dbm_duration_seconds, which stays accurate across process crashes and restarts; kxi_sm_dbm_discontinued_count, how often migrations are interrupted; and kxi_sm_dbm_pending_partitions, the number of partitions waiting to be migrated.

  • A new guide on Interpreting DBM Metrics has been added to help better understand the states of a database migration.

  • A new opt-in, low-memory sort optimization is available in the Storage Manager, controlled by the KXI_SM_SKIP_SORT_CHECK environment variable. By default (false), the Storage Manager checks whether the source is already sorted and can be copied without re-sorting during end-of-interval writedown and batch ingest. Set it to true to skip this check for tables with more than one sort column, where the check is most expensive, and re-sort the source table instead.

  • A new environment variable, KXI_VALIDATION_MAX_PARTITIONS, caps how many date partitions are checked during object-store validation. See Database validation for more details.

  • Query responses returned through the Service Gateway now include additional fields identifying which Gateway, Resource Coordinator and Aggregator instance handled the request. This makes it easier to trace and troubleshoot a specific query's path through the system, particularly in multi-instance deployments.

  • The Get Data Query API now supports the use of the date column for the groupBy parameter.

Web Interface

  • Pipeline Settings now include an Annotations section for applying annotations to a pipeline. This enables service account authentication when using the S3 or Azure file reader.

  • The Database Reader node in the Pipeline Editor now has additional parameters: limit to cap the number of rows returned, scope to add routing configuration so the query is directed to the appropriate package, tier and DAP, inputTZ, outputTZ and outputTZCols for timezone handling, fill for missing values, and temporality. UDAs now also support the top-level scope parameter. See Readers for more details.

Stream Processor

  • New latency profiling exposes probes you can use to identify performance issues and bottlenecks in a pipeline. This improves the capabilities for Debugging pipelines when inspecting data flow, payloads and latency within a running pipeline.

  • Reader operators in Python pipelines now support environment variables through sp.use_var(), so reader parameters can be supplied from Kubernetes secrets and configmaps rather than hard-coded in the pipeline.

  • The v2 file readers for q, Python and the Web Interface now accept a readiness hook, letting you hold back processing until the files in a bucket are ready. This helps in cases where files are rewritten before they are final, or where a separate trigger file signals that a batch is complete. The hook applies to watching mode only; with a one-shot reader, delay starting the pipeline instead.

Package Manager

  • kxi pm add dep <dependency> adds a dependency or module to a package that has already been pushed, without pulling the package locally, editing it and pushing it back.

  • The kxi package field command now has a delete option for removing a field from a YAML file.

Scratchpad
  • Readers that support authentication through service accounts (AWS S3 and MS Azure) are now supported in pipeline quick tests and Scratchpad sessions, so a pipeline that relies on service account authentication can be tested without deploying it.
Command Line Interface

  • New kxi group commands manage the users and roles in a group from the CLI. Previously you could assign roles to individual users and add users to groups, but assigning a role directly to a group required calling the Keycloak API.

  • kxi user get-effective-roles returns the effective roles for a user, including those inherited from the groups they belong to.

  • kxi profile list lists the profiles you have set up, showing the name, namespace, hostname, usage, realm and encryption fields.

Fixes

kdb Insights Enterprise 1.20 resolves a number of issues across the product. Storage Manager fixes cover database conversion, end-of-interval handling and schema validation on ingest. Stream Processor fixes address the Python S3 writer and environment variable handling, and there are further fixes for entitlement propagation, metrics collection and the CLI. Expand the following sections for more details.

Database

  • Improved Storage Manager recovery after an interrupted database conversion. Previously, a conversion that hit a disk space quota could leave the Storage Manager in an inconsistent state.

  • Fixed issues combining certain getData arguments, notably groupBy and fill, when they are specified as empty.

  • Fixed the query error message returned when an API or table does not exist on the specified scope.

  • The end-of-interval (EOI) process no longer restarts indefinitely after an initial-import database validation failure. EOI now stays alive in an error state and notifies the Storage Manager, so you can restart it manually once the underlying data issue is resolved.

  • Compound columns now retain their correct type during a database conversion, rather than being converted to string.

  • The Storage Manager now validates the schema of ingested empty tables. Previously, an empty table published with the wrong schema could give IDB tables incorrect column types and cause the next end-of-day writedown to fail.

  • The Storage Manager now reduces the chunk size on schema conversions when running multithreaded.

  • Fixed sorting of HDB tiers by the index property, which now reorders all storage tiers rather than only the HDB.

Stream Processor

  • .qsp.getSchema now takes an optional second argument specifying which mounted database config file (configmap) to parse the schema from, so pipelines with multiple assemblies mounted in the config directory resolve the intended schema.

  • Fixed the S3 writer node failing in Python, although it worked as expected in q.

  • Fixed an issue where a pipeline rejected integer-typed node parameters set as environment variables using sp.use_var().

Package Manager
  • Fixed an issue where deploying a single pipeline using the CLI redeployed the rest of the package's entities, restarting the whole assembly.
Entitlements
  • Deleting a group in Keycloak is now fully propagated to the Entitlements Service. Previously, the deletion could leave stale entitlement mappings behind, so if the group was later recreated under the same name, its users were not displayed against it and did not pick up the correct entitlements.
Monitoring
  • kxi monitoring install and kxi monitoring uninstall now correctly remove and reinstate the prometheus-operator-crds chart.
Command Line Interface

  • kxi query can now run SQL against production, not just against Query Environments. Previously, SQL was unavailable from the CLI when the Query Environment was disabled.

  • When the API gateway is unavailable, the CLI now reports that it is retrying instead of appearing to hang. The message is shown even without debug mode enabled. For example, "API gateway not ready (3/60). Retrying in 5s".

Install/Upgrade
  • The cluster issuer is now updated to match the ingress controller type, so HTTP validation succeeds.
Security
  • Various CVEs were remediated as part of this release.

Deprecations

  • The v1 Stream Reader, v1 Stream Writer and v1 Database Writer are deprecated. Move pipelines to the v2 nodes, which support stream-side topic filtering and per-node stream configuration.

  • The kxi-python dependency in the Stream Processor is deprecated and will be removed in a future release.

Third-party Dependencies

kdb Insights Enterprise 1.20.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.20.0 release of the standalone infrastructure installation scripts.

For important information on supported versions, see the following pages:

Artifacts

Type Downloads Portal location
Enterprise insights-1.20.0.tgz
Operator kxi-operator-1.20.0.tgz
CLI kxicli-1.20.0-py3-none-any.whl
RT q interface rt.1.20.0.qpk
RT C interface kxi-c-sdk 1.20.0
RT Python interface kxi-rtpy-1.19.0
RT Java interface kxi-java-sdk 1.16.0
RT C# interface kxi-csharp-sdk 1.18.0
Infrastructure kxi-terraform-1.20.0.tgz
kxi-management-service kxi-management-service-1.1.8.tgz
insights-on-k8s insights-on-k8s-1.1.51.tgz

We look forward to bringing you even bigger features in kdb Insights Enterprise, coming soon!

Back to top