kdb Insights SDK - Release Notes¶

This page details the release notes for the latest series of kdb Insights SDK releases. To find the release notes for an older release, refer to the previous releases page.

1.20.0¶

Release date: 2026-09-21

kdb Insights SDK 1.20 focuses on pipeline manageability and operational visibility. It introduces granular status tracking for long-running pipelines, an in-memory only table type, resilience and observability improvements in Reliable Transport, and new metrics for monitoring database tier migrations.

Breaking Changes and Upgrade Notes

Reliable Transport Publisher Logs

RT publishers no longer need to retain their local client logs between restarts. Because publisher state is now tracked differently, you cannot move directly between 1.18 and 1.20 in either direction: both upgrades and rollbacks must be staged through 1.19.

Insights Nexus Download Portal Retired

The nexus.dl.kx.com download site has been decommissioned. All Helm charts, images, wheels and raw artifacts are served from the KX Downloads Portal. Update any install scripts, CI pipelines or private mirrors still pointing at the Nexus host.

New Features¶

In-memory only tables

A new table type, mem_only, defines a table that resides solely in the RDB with no disk persistence. Data survives normal end-of-interval and end-of-day cycles, and is cleared only when the process restarts.

Note

Because nothing is written to disk, the table grows for the lifetime of the process. Do not use this table type for tables whose data volume grows over time.
Granular Pipeline Status Tracking

Pipelines now report status for each individual unit, and branches within a pipeline can be triggered independently. This means work that previously needed separate pipelines can be consolidated into one long-running pipeline and sequenced internally.

Consolidating pipelines this way reduces the number of persistent volumes a deployment needs, because each pipeline previously required at least two. It also removes the startup, shutdown and disk attach/detach latency of a start/stop scheduling model.

Refer to Pipeline Units for more details.
RT Publisher Resilience

An RT publisher that has lost or deleted its local log directory now reconnects and resumes publishing from where it left off. Previously, restarting a publisher whose RT logs had been removed caused RT to reject the connection, and no new data reached the stream.

Note

This changes the supported upgrade path. See Breaking Changes and Upgrade Notes at the top of this release for more details.
RT Observability

A new API, /rt-clients-meta, returns sequencer state information from Reliable Transport, including the state held for a given publisher. This gives you a supported way to inspect stream position and publisher registration when inspecting your data ingestion stream.

Improvements¶

1.20 brings a broad set of improvements across the product. Database work focuses on observability, with new metrics for database tier migrations, identification of the Gateway, Resource Coordinator and Aggregator that served a query, and an opt-in low-memory sort optimization. The Stream Processor adds latency profiling, environment variable support in Python reader operators, and a readiness hook for the v2 file readers. Expand the following sections for more details.

Database

A new Storage Manager metric, kxi_sm_dbm_status, exposes the status of DBM tier migrations. The gauge reports statuses of idle, active, or paused when a migration has been suspended due to an EOD or a DBM process restart. A paused migration resumes automatically once the interrupting process completes.
Three additional DBM metrics improve visibility into DBM health and backlog: kxi_sm_dbm_duration_seconds, which stays accurate across process crashes and restarts; kxi_sm_dbm_discontinued_count, how often migrations are interrupted; and kxi_sm_dbm_pending_partitions, the number of partitions waiting to be migrated.
A new guide on Interpreting DBM Metrics has been added to help better understand the states of a database migration.
A new opt-in, low-memory sort optimization is available in the Storage Manager, controlled by the KXI_SM_SKIP_SORT_CHECK environment variable. By default (false), the Storage Manager checks whether the source is already sorted and can be copied without re-sorting during end-of-interval writedown and batch ingest. Set it to true to skip this check for tables with more than one sort column, where the check is most expensive, and re-sort the source table instead.
A new environment variable, KXI_VALIDATION_MAX_PARTITIONS, caps how many date partitions are checked during object-store validation. See Initial Import Process for more details.
Query responses returned through the Service Gateway now include additional fields identifying which Gateway, Resource Coordinator and Aggregator instance handled the request. This makes it easier to trace and troubleshoot a specific query's path through the system, particularly in multi-instance deployments.
The Get Data Query API now supports the use of the date column for the groupBy parameter.

Stream Processor

New latency profiling exposes probes you can use to identify performance issues and bottlenecks in a pipeline. This improves the capabilities for troubleshooting pipelines when inspecting data flow, payloads and latency within a running pipeline.
Reader operators in Python pipelines now support environment variables through sp.use_var(), so reader parameters can be supplied from Kubernetes secrets and configmaps rather than hard-coded in the pipeline.
The v2 file readers for q and Python now accept a readiness hook, letting you hold back processing until the files in a bucket are ready. This helps in cases where files are rewritten before they are final, or where a separate trigger file signals that a batch is complete. The hook applies to watching mode only; with a one-shot reader, delay starting the pipeline instead.

Fixes¶

1.20 resolves a number of issues across the product. Storage Manager fixes cover database conversion, end-of-interval handling and schema validation on ingest. Stream Processor fixes address the Python S3 writer, schema resolution and environment variable handling. Expand the following sections for more details.

Database

Improved Storage Manager recovery after an interrupted database conversion. Previously, a conversion that hit a disk space quota could leave the Storage Manager in an inconsistent state.
Fixed issues combining certain getData arguments, notably groupBy and fill, when they are specified as empty.
Fixed the query error message returned when an API or table does not exist on the specified scope.
The end-of-interval (EOI) process no longer restarts indefinitely after an initial-import database validation failure. EOI now stays alive in an error state and notifies the Storage Manager, so you can restart it manually once the underlying data issue is resolved.
Compound columns now retain their correct type during a database conversion, rather than being converted to string.
The Storage Manager now validates the schema of ingested empty tables. Previously, an empty table published with the wrong schema could give IDB tables incorrect column types and cause the next end-of-day writedown to fail.
The Storage Manager now reduces the chunk size on schema conversions when running multithreaded.
Fixed sorting of HDB tiers by the index property, which now reorders all storage tiers rather than only the HDB.

Stream Processor

.qsp.getSchema now takes an optional second argument specifying which mounted database config file (configmap) to parse the schema from, so pipelines with multiple assemblies mounted in the config directory resolve the intended schema.
Fixed the S3 writer node failing in Python, although it worked as expected in q.
Fixed an issue where a pipeline rejected integer-typed node parameters set as environment variables using sp.use_var().

Security

Various CVEs were remediated as part of this release.

Deprecations¶

The v1 Stream Reader, v1 Stream Writer and v1 Database Writer are deprecated. Move pipelines to the v2 nodes, which support stream-side topic filtering and per-node stream configuration.
The kxi-python dependency in the Stream Processor is deprecated and will be removed in a future release.

Artifacts¶

All artifacts updated for this release are highlighted.

Stream Processor¶

service	Downloads Portal Docker
Controller	portal.dl.kx.com/kxi-sp-controller:1.20.0
Worker	portal.dl.kx.com/kxi-sp-worker:1.20.0
Coordinator	portal.dl.kx.com/kxi-sp-coordinator:1.20.0
Worker-Python	portal.dl.kx.com/kxi-sp-python:1.20.0
Worker-ML	portal.dl.kx.com/kxi-ml:1.20.0

Database¶

service	Downloads Portal Docker	Helm Chart
Data Access Process (DAP)	portal.dl.kx.com/kxi-da:1.20.0
Single container	portal.dl.kx.com/kxi-da-single:1.20.0
Storage Manager	portal.dl.kx.com/kxi-sm-single:1.20.0
Gateway	portal.dl.kx.com/kxi-sg-gw:1.20.0
Resource Coordinator	portal.dl.kx.com/kxi-sg-rc:1.20.0
Aggregator	portal.dl.kx.com/kxi-sg-agg:1.20.0
Rollback	portal.dl.kx.com/kxi-db-rollback:1.20.0

Reliable Transport¶

service	Downloads Portal Docker	Helm Chart
Reliable Transport	portal.dl.kx.com/kxi-rt:1.20.0	kxi-rt-1.20.0.tgz

Miscellaneous¶

service	Downloads Portal Docker
Metrics Sidecar	portal.dl.kx.com/kxi-sidecar:1.20.0
OpenAPI SDK Generator	portal.dl.kx.com/kxi-openapi-codegen:0.8.0