Skip to content

kdb Insights Microservices - Release Notes

This page details the release notes for the latest series of kdb Insights Microservices releases. To find the release notes for an older release, refer to the previous releases page.

1.12.7

Release Date 2025-04-17

Fixes

Expand to see the full details of resolved issues

Database

  • Resolved an issue where the selectTable API could return an incorrect result if executing an aggregation that included data on disk. For example, a call such as .kxi.selectTabletableagg!(trade;enlist[x]!enlist(avg;price))` could be subject to this defect.

Stream Processor

  • Resolved a memory leak that occurred when using the MQTT reader.

Security

  • Various CVEs were remediated as part of this release.

Artifacts

KX Nexus

The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users.

Stream Processor

Service Nexus Docker Downloads Portal Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.12.7 portal.dl.kx.com/kxi-sp-controller:1.12.7
Worker registry.dl.kx.com/kxi-sp-worker:1.12.7 portal.dl.kx.com/kxi-sp-worker:1.12.7
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.12.7 portal.dl.kx.com/kxi-sp-coordinator:1.12.7
Worker-Python registry.dl.kx.com/kxi-sp-python:1.12.7 portal.dl.kx.com/kxi-sp-python:1.12.7
Worker-ML registry.dl.kx.com/kxi-ml:1.12.7 portal.dl.kx.com/kxi-ml:1.12.7

Database

Service Nexus Docker Downloads Portal Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.12.7 portal.dl.kx.com/kxi-da:1.12.7
Single container registry.dl.kx.com/kxi-da-single:1.12.7 portal.dl.kx.com/kxi-da-single:1.12.7
Storage Manager registry.dl.kx.com/kxi-sm-single:1.12.7 portal.dl.kx.com/kxi-sm-single:1.12.7
Gateway registry.dl.kx.com/kxi-sg-gw:1.12.7 portal.dl.kx.com/kxi-sg-gw:1.12.7
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.12.7 portal.dl.kx.com/kxi-sg-rc:1.12.7
Aggregator registry.dl.kx.com/kxi-sg-agg:1.12.7 portal.dl.kx.com/kxi-sg-agg:1.12.7
Rollback registry.dl.kx.com/kxi-db-rollback:1.12.7 portal.dl.kx.com/kxi-db-rollback:1.12.7

Reliable Transport

Service Nexus Docker Downloads Portal Docker Helm Chart
Reliable Transport registry.dl.kx.com/kxi-rt:1.12.5 portal.dl.kx.com/kxi-rt:1.12.5 kxi-rt-1.12.5.tgz

Miscellaneous

Service Nexus Docker Package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.12.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

1.12.6

Release Date 2025-03-25

Fixes

Expand to see the full details of resolved issues

Database

  • Schema change performance has been improved to support multi-threaded operations, and to only execute against only the impacted tables.

  • The database now waits to initialize connection to Reliable Transport until its processes are fully online. Previously, DAPs would sometimes try to connect before fully available, resulting in an excessive number of RT logs being generated and issues with their processing.

Reliable Transport

  • Resolved an issue in which an incorrect RT_LOG_PATH was set after a restart by qsup.

Security

  • Various CVEs were remediated as part of this release.

Artifacts

KX Nexus

The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users.

Stream Processor

Service Nexus Docker Downloads Portal Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.12.6 portal.dl.kx.com/kxi-sp-controller:1.12.6
Worker registry.dl.kx.com/kxi-sp-worker:1.12.6 portal.dl.kx.com/kxi-sp-worker:1.12.6
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.12.6 portal.dl.kx.com/kxi-sp-coordinator:1.12.6
Worker-Python registry.dl.kx.com/kxi-sp-python:1.12.6 portal.dl.kx.com/kxi-sp-python:1.12.6
Worker-ML registry.dl.kx.com/kxi-ml:1.12.6 portal.dl.kx.com/kxi-ml:1.12.6

Database

Service Nexus Docker Downloads Portal Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.12.6 portal.dl.kx.com/kxi-da:1.12.6
Single container registry.dl.kx.com/kxi-da-single:1.12.6 portal.dl.kx.com/kxi-da-single:1.12.6
Storage Manager registry.dl.kx.com/kxi-sm-single:1.12.6 portal.dl.kx.com/kxi-sm-single:1.12.6
Gateway registry.dl.kx.com/kxi-sg-gw:1.12.6 portal.dl.kx.com/kxi-sg-gw:1.12.6
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.12.6 portal.dl.kx.com/kxi-sg-rc:1.12.6
Aggregator registry.dl.kx.com/kxi-sg-agg:1.12.6 portal.dl.kx.com/kxi-sg-agg:1.12.6
Rollback registry.dl.kx.com/kxi-db-rollback:1.12.6 portal.dl.kx.com/kxi-db-rollback:1.12.6

Reliable Transport

Service Nexus Docker Downloads Portal Docker Helm Chart
Reliable Transport registry.dl.kx.com/kxi-rt:1.12.5 portal.dl.kx.com/kxi-rt:1.12.5 kxi-rt-1.12.5.tgz

Miscellaneous

Service Nexus Docker Package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.12.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

1.12.5

Release Date 2025-03-12

Fixes

Expand to see the full details of resolved issues

Insights Core

  • Insights 1.12.5 has been updated to run Insights Core 4.1.10, which contains fixes related to the object storage library. Refer to the Insights Core release notes for details of these fixes.

Security

  • Various CVEs were remediated as part of this release.

Artifacts

KX Nexus

The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users.

Stream Processor

Service Nexus Docker Downloads Portal Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.12.5 portal.dl.kx.com/kxi-sp-controller:1.12.5
Worker registry.dl.kx.com/kxi-sp-worker:1.12.5 portal.dl.kx.com/kxi-sp-worker:1.12.5
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.12.5 portal.dl.kx.com/kxi-sp-coordinator:1.12.5
Worker-Python registry.dl.kx.com/kxi-sp-python:1.12.5 portal.dl.kx.com/kxi-sp-python:1.12.5
Worker-ML registry.dl.kx.com/kxi-ml:1.12.5 portal.dl.kx.com/kxi-ml:1.12.5

Database

Service Nexus Docker Downloads Portal Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.12.5 portal.dl.kx.com/kxi-da:1.12.5
Single container registry.dl.kx.com/kxi-da-single:1.12.5 portal.dl.kx.com/kxi-da-single:1.12.5
Storage Manager registry.dl.kx.com/kxi-sm-single:1.12.5 portal.dl.kx.com/kxi-sm-single:1.12.5
Gateway registry.dl.kx.com/kxi-sg-gw:1.12.5 portal.dl.kx.com/kxi-sg-gw:1.12.5
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.12.5 portal.dl.kx.com/kxi-sg-rc:1.12.5
Aggregator registry.dl.kx.com/kxi-sg-agg:1.12.5 portal.dl.kx.com/kxi-sg-agg:1.12.5
Rollback registry.dl.kx.com/kxi-db-rollback:1.12.5 portal.dl.kx.com/kxi-db-rollback:1.12.5

Reliable Transport

Service Nexus Docker Downloads Portal Docker Helm Chart
Reliable Transport registry.dl.kx.com/kxi-rt:1.12.5 portal.dl.kx.com/kxi-rt:1.12.5 kxi-rt-1.12.5.tgz

Miscellaneous

Service Nexus Docker Package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.12.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

1.12.4

Release Date 2025-03-05

Fixes

Expand to see the full details of resolved issues

Database

  • Executing a .kxi.sql2 or .kxi.getData query and only suppling table name and limit in the arguments caused an out-of-memory error for large databases. Now, the limit is applied (if possible) before unenumerating the data.

  • Temporary files created by NFS were preventing Storage Manager from deleting directories during on-disk sort. This has been resolved.

Reliable Transport

  • When subscribing to the latest position in RT, all log files, rather than just the log files with the latest data, were copied to the subscriber node. Now, only the log files required are copied.

  • RT copies log files from the RT node to the client node. If a log file is truncated by RT on the RT node, it gets truncated on the subscriber node. The subscriber can still truncate the file earlier than RT, but now RT will truncate the file on the client, even if the client has opted out of the RT garbage collection. This mitigates problems that could arise if a client keeps log files longer than RT.

Artifacts

KX Nexus

The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users.

Stream Processor

Service Nexus Docker Downloads Portal Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.12.4 portal.dl.kx.com/kxi-sp-controller:1.12.4
Worker registry.dl.kx.com/kxi-sp-worker:1.12.4 portal.dl.kx.com/kxi-sp-worker:1.12.4
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.12.4 portal.dl.kx.com/kxi-sp-coordinator:1.12.4
Worker-Python registry.dl.kx.com/kxi-sp-python:1.12.4 portal.dl.kx.com/kxi-sp-python:1.12.4
Worker-ML registry.dl.kx.com/kxi-ml:1.12.4 portal.dl.kx.com/kxi-ml:1.12.4

Database

Service Nexus Docker Downloads Portal Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.12.4 portal.dl.kx.com/kxi-da:1.12.4
Single container registry.dl.kx.com/kxi-da-single:1.12.4 portal.dl.kx.com/kxi-da-single:1.12.4
Storage Manager registry.dl.kx.com/kxi-sm-single:1.12.4 portal.dl.kx.com/kxi-sm-single:1.12.4
Gateway registry.dl.kx.com/kxi-sg-gw:1.12.4 portal.dl.kx.com/kxi-sg-gw:1.12.4
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.12.4 portal.dl.kx.com/kxi-sg-rc:1.12.4
Aggregator registry.dl.kx.com/kxi-sg-agg:1.12.4 portal.dl.kx.com/kxi-sg-agg:1.12.4
Rollback registry.dl.kx.com/kxi-db-rollback:1.12.4 portal.dl.kx.com/kxi-db-rollback:1.12.4

Reliable Transport

Service Nexus Docker Downloads Portal Docker Helm Chart
Reliable Transport registry.dl.kx.com/kxi-rt:1.12.4 portal.dl.kx.com/kxi-rt:1.12.4 kxi-rt-1.12.4.tgz

Miscellaneous

Service Nexus Docker Package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.12.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

1.12.3

Release Date 2025-02-05

Fixes

Expand to see the full detail of resolved issues here

Database

  • .kxi.getData would hit a length error when trying to apply more than one filter to a reference table joined via a foreign key.

  • If executing a .kxi.selectTable query in the RDB for a partitioned table, and the startTS and endTS did not intersect with the purview of the RDB DAP, all data for the table would be returned. This has been addressed so it will now return an empty table if there is no overlap within the purview.

Artifacts

KX Nexus

The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users.

Stream Processor

Service Nexus Docker Downloads Portal Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.12.2 portal.dl.kx.com/kxi-sp-controller:1.12.2
Worker registry.dl.kx.com/kxi-sp-worker:1.12.2 portal.dl.kx.com/kxi-sp-worker:1.12.2
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.12.2 portal.dl.kx.com/kxi-sp-coordinator:1.12.2
Worker-Python registry.dl.kx.com/kxi-sp-python:1.12.2 portal.dl.kx.com/kxi-sp-python:1.12.2
Worker-ML registry.dl.kx.com/kxi-ml:1.12.2 portal.dl.kx.com/kxi-ml:1.12.2

Database

Service Nexus Docker Downloads Portal Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.12.3 portal.dl.kx.com/kxi-da:1.12.3
Single container registry.dl.kx.com/kxi-da-single:1.12.3 portal.dl.kx.com/kxi-da-single:1.12.3
Storage Manager registry.dl.kx.com/kxi-sm-single:1.12.3 portal.dl.kx.com/kxi-sm-single:1.12.3
Gateway registry.dl.kx.com/kxi-sg-gw:1.12.3 portal.dl.kx.com/kxi-sg-gw:1.12.3
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.12.3 portal.dl.kx.com/kxi-sg-rc:1.12.3
Aggregator registry.dl.kx.com/kxi-sg-agg:1.12.3 portal.dl.kx.com/kxi-sg-agg:1.12.3
Rollback registry.dl.kx.com/kxi-db-rollback:1.12.3 portal.dl.kx.com/kxi-db-rollback:1.12.3

Reliable Transport

Service Nexus Docker Downloads Portal Docker Helm Chart
Reliable Transport registry.dl.kx.com/kxi-rt:1.12.0 portal.dl.kx.com/kxi-rt:1.12.0 kxi-rt-1.12.0.tgz

Miscellaneous

Service Nexus Docker Package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.12.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

1.12.2

Release Date 2025-01-30

Improvements

Database

  • The helper function .kxi.selectTable now supports the following:

    • The limit parameter. When no aggregation is required .kxi.sql2 and .kxi.getData apply the limit parameter closer to the raw data selection, improving performance of the query.
    • Filters that use the virtual column i.

  • When using User-Defined Analytics (UDAs), the userName value is now available when the helper function .kxi.getHeader is called in the DAP. See response header for details on all fields of the response header, and kxi.getHeader for an example.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Temporary files created by NFS were preventing Storage Manager from deleting directories during post-commit cleanup. This has been resolved through improved handling of .nfs files and changes to how directory deletion is performed.

  • Resolved an issue in which the Service Gateway was not registering new services when they were deployed, resulting in queries failing.

Artifacts

KX Nexus

The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users.

Stream Processor

service Nexus Docker Downloads Portal Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.12.2 portal.dl.kx.com/kxi-sp-controller:1.12.2
Worker registry.dl.kx.com/kxi-sp-worker:1.12.2 portal.dl.kx.com/kxi-sp-worker:1.12.2
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.12.2 portal.dl.kx.com/kxi-sp-coordinator:1.12.2
Worker-Python registry.dl.kx.com/kxi-sp-python:1.12.2 portal.dl.kx.com/kxi-sp-python:1.12.2
Worker-ML registry.dl.kx.com/kxi-ml:1.12.2 portal.dl.kx.com/kxi-ml:1.12.2

Database

service Nexus Docker Downloads Portal Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.12.2 portal.dl.kx.com/kxi-da:1.12.2
Single container registry.dl.kx.com/kxi-da-single:1.12.2 portal.dl.kx.com/kxi-da-single:1.12.2
Storage Manager registry.dl.kx.com/kxi-sm-single:1.12.2 portal.dl.kx.com/kxi-sm-single:1.12.2
Gateway registry.dl.kx.com/kxi-sg-gw:1.12.2 portal.dl.kx.com/kxi-sg-gw:1.12.2
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.12.2 portal.dl.kx.com/kxi-sg-rc:1.12.2
Aggregator registry.dl.kx.com/kxi-sg-agg:1.12.2 portal.dl.kx.com/kxi-sg-agg:1.12.2
Rollback registry.dl.kx.com/kxi-db-rollback:1.12.2 portal.dl.kx.com/kxi-db-rollback:1.12.2

Reliable Transport

service Nexus Docker Downloads Portal Docker Helm Chart
Reliable Transport registry.dl.kx.com/kxi-rt:1.12.0 portal.dl.kx.com/kxi-rt:1.12.0 kxi-rt-1.12.0.tgz

Miscellaneous

service Nexus Docker package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.12.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

1.12.1

Release Date 2025-01-21

Improvements

Database

  • Improvements have been made in Storage Manager to reduce the time spent on migrating data across tiers (when compression settings across tiers are the same).
  • A new optional environment variable KXI_SKIP_HDB_SIZE has been introduced to bypass the HDB size calculation during startup. When enabled, this variable skips the calculation of HDB sizes during SM initialization which is beneficial for systems with large HDBs, where the size calculation can be time-consuming. See Database configuration for details on setting this in your assembly's YAML file.
  • When using User-Defined Analytics (UDAs), the userID value is now available when the helper function .kxi.getHeader is called in the DAP. See response header for details on all fields of the response header, and kxi.getHeader for an example.

Python Pipelines

  • Python pipelines now support reading from databases using User-Defined Analytics (UDAs). For more information, refer to the Python readers documentation.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Resolved an issue where late data added to an existing HDB table partition could create duplicate records at EOD write-down if the memory available to the Storage Manager process is very small.

  • Resolved an issue where the KXI_SM_HDB_SIZE metric caused delays in SM initialization for systems with large historical databases (HDBs). See Environment Variables for details on this metric.

  • Storage Manager now catches attempted column type conversions that aren't supported and provides a more helpful log message before shutting down.

  • When executing a batch ingest that spans multiple days, the Storage Manager now performs RAM garbage collection between dates. This avoids the potential for out-of-memory failures.

Artifacts

KX Nexus

The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users.

Stream Processor

service Nexus Docker Downloads Portal Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.12.1 portal.dl.kx.com/kxi-sp-controller:1.12.1
Worker registry.dl.kx.com/kxi-sp-worker:1.12.1 portal.dl.kx.com/kxi-sp-worker:1.12.1
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.12.1 portal.dl.kx.com/kxi-sp-coordinator:1.12.1
Worker-Python registry.dl.kx.com/kxi-sp-python:1.12.1 portal.dl.kx.com/kxi-sp-python:1.12.1
Worker-ML registry.dl.kx.com/kxi-ml:1.12.1 portal.dl.kx.com/kxi-ml:1.12.1

Database

service Nexus Docker Downloads Portal Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.12.1 portal.dl.kx.com/kxi-da:1.12.1
Single container registry.dl.kx.com/kxi-da-single:1.12.1 portal.dl.kx.com/kxi-da-single:1.12.1
Storage Manager registry.dl.kx.com/kxi-sm-single:1.12.1 portal.dl.kx.com/kxi-sm-single:1.12.1
Gateway registry.dl.kx.com/kxi-sg-gw:1.12.1 portal.dl.kx.com/kxi-sg-gw:1.12.1
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.12.1 portal.dl.kx.com/kxi-sg-rc:1.12.1
Aggregator registry.dl.kx.com/kxi-sg-agg:1.12.1 portal.dl.kx.com/kxi-sg-agg:1.12.1
Rollback registry.dl.kx.com/kxi-db-rollback:1.12.1 portal.dl.kx.com/kxi-db-rollback:1.12.1

Reliable Transport

service Nexus Docker Downloads Portal Docker Helm Chart
Reliable Transport registry.dl.kx.com/kxi-rt:1.12.0 portal.dl.kx.com/kxi-rt:1.12.0 kxi-rt-1.12.0.tgz

Miscellaneous

service Nexus Docker package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.12.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

1.12.0

Release Date 2024-12-11

kdb Insights SDK 1.12 introduces a number of new features and improvements, including; an enhanced data import and ingest experience, the ability to run Reliable Transport outside of a container, and more. Read on to find out more!

New Features

  1. Batch Ingest Features & Enhancements
  2. Initial Database Import
  3. Natively define Python User-Defined Analytics (UDAs)
  4. Reliable Transport on Bare Metal Hardware
  5. Support calling UDAs from Pipelines

1. Batch Ingest Features & Enhancements

This release introduces a new version of the database writer, which improves the batch ingest experience when using direct write. The new database writer includes:

  • Support for partitioned pipelines, allowing improved performance using parallelization. When maxWorkers is set to more than 1, the Stream Processor (SP) scales up to multiple workers, each taking a shard of the workload and processing it in parallel. This means a much faster processing time.
  • Support for unbounded readers, allowing pipelines to read continuous streams of data but still use direct write to the database. Examples of unbounded use cases include file readers with file watching enabled, stream readers (Kafka, Insights Streams, MQTT), callback readers, or upload readers. This support removes the restrictions and limitations around the support of these use cases.
  • Full support in both Insights SDK deployments of the Stream Processor in both Docker and Kubernetes.
  • Monitoring of database ingest sessions using a new API. This allows you to monitor the state of ingest sessions by returning a list of them and their various states: pending, queued, progressing, completed or errored.

An overview of the database ingest functionality is available here, and the API details can be found here: Python and q.

See the walkthrough for additional guidance.

2. Initial Database Import Enhancements

You can import your existing kdb+ database into kdb Insights SDK to seed your empty database with data. This release improves upon the existing the process, and also provides:

  • Embedded validation to ensure the alignment of the schemas and the appropriate attributes
  • Graceful handling of failures with a supported way to re-try/re-start ingest if it does fail for some reason

Read the quickstart guide for a walkthrough of using the initial import feature. The full import process can be followed here.

3. Natively define Python User-Defined Analytics (UDAs)

You can now develop your UDAs natively in Python. Read the Python API documentation regarding UDAs for more details.

4. Reliable Transport on Bare Metal Hardware

Reliable Transport (RT) can now be run on bare metal hardware, with no containerization required. Read this walkthrough for details.

5. Support calling UDAs from Pipelines

UDAs can now be queried from pipelines using the database reader. Read the q and Python APIs documentation for how to take advantage of this feature.

Improvements

Audit Logging

The following operations now have audit logging to offer traceability of key system operations:

  • Queries - the entirety of the query path now provides an audit-id that makes tracing your query across processes quick and easy.
  • Stream Processor - Correlation IDs are now logged by the audit framework when using an SP upload reader.

New SQL Operators

  • SQL2 now supports the following new operators:
    • reverse
    • char_length
    • character_length
    • bit_length
    • length
    • octet_length

Parquet Reader

  • A local file watcher has been added for the Parquet reader. Read the q and Python documentation for details.

  • The Parquet reader now allows you to specify the file path as a symbol.

Stream Processor

  • The Python API has been updated to include .qsp.track functionality that the q API already supports. Read the track page for details.

Database Column Re-sorting

  • It is now possible to change the defined sort column of a table(s) for an existing database. This is an expensive, long-running operation for existing, large-scale databases. This should be considered before execution. For this reason, it is not enabled by default. To use, set the environment variable KXI_SM_ALLOW_CHANGE_SORT_COLS=1 in Storage Manager and then follow the steps for modifying a schema.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Resolved an issue in which Storage Manager's RAM requirements increased for batch ingest operations. This has been returned to resource levels in line with release 1.10.

  • Previously, a failed end-of-day writedown (EOD) was not retried by the Storage Manager, causing a misaligned error on the next EOD.

  • Fixed an issue when the Database was not being identified as having unsaved changes when primary keys were changed.

  • Fixed an issue where DAP code assumed the values of assembly mount names was one of "RDB", "IDB", or "HDB". This has been resolved to allow for mount names of any description.

  • Resolved an issue where a SQL2 query would have produced a type error when using a GROUP BY statement with an ORDER BY of a subset of the fields with no aggregation. For example, select sym,realTime from trade group by sym, realTime order by sym.

  • Resolved an issue where a SQL2 query would have produced a type error when using a GROUP BY statement with a DISTINCT and no other aggregation.

  • Resolved an issue where a query for an empty date could fail when late data is turned on.

  • Previously, if a Stream Processor published a message to a subscriber node that contained a null float, and the WebSocket client reads the messages in JSON, the Service Gateway would error with NaN is not a valid double value as per JSON specification error. This has been resolved.

Readers

  • Using a Parquet reader with a glob pattern would error when trying to find or read a file. This has been resolved.

  • Fixed an edge-case issue where a length error would occur Failed to initialize worker, worker name=[sample name] error=length when using glob patterns with file names.

  • The discrepancy in how AWS and Azure Cloud readers retained an extra newline character when reading in text mode without chunking, while the Google Cloud reader correctly did not, has now been resolved.

  • When establishing a secret used for GCP authentication for a Parquet reader, a pipeline would fail with a Worker failed to start: Unable to locate credentials secret error.

File Watchers

  • Passing in a dict for the watch parameter was not working correctly. Furthermore, use of the additional options the web interface provides when Use Watching tick box meant a dict was used, and therefore also failed. This has been resolved.

Artifacts

KX Nexus

The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users.

Stream Processor

service Nexus Docker Downloads Portal Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.12.0 portal.dl.kx.com/kxi-sp-controller:1.12.0
Worker registry.dl.kx.com/kxi-sp-worker:1.12.0 portal.dl.kx.com/kxi-sp-worker:1.12.0
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.12.0 portal.dl.kx.com/kxi-sp-coordinator:1.12.0
Worker-Python registry.dl.kx.com/kxi-sp-python:1.12.0 portal.dl.kx.com/kxi-sp-python:1.12.0
Worker-ML registry.dl.kx.com/kxi-ml:1.12.0 portal.dl.kx.com/kxi-ml:1.12.0

Database

service Nexus Docker Downloads Portal Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.12.0 portal.dl.kx.com/kxi-da:1.12.0
Single container registry.dl.kx.com/kxi-da-single:1.12.0 portal.dl.kx.com/kxi-da-single:1.12.0
Storage Manager registry.dl.kx.com/kxi-sm-single:1.12.0 portal.dl.kx.com/kxi-sm-single:1.12.0
Gateway registry.dl.kx.com/kxi-sg-gw:1.12.0 portal.dl.kx.com/kxi-sg-gw:1.12.0
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.12.0 portal.dl.kx.com/kxi-sg-rc:1.12.0
Aggregator registry.dl.kx.com/kxi-sg-agg:1.12.0 portal.dl.kx.com/kxi-sg-agg:1.12.0
Rollback registry.dl.kx.com/kxi-db-rollback:1.12.0 portal.dl.kx.com/kxi-db-rollback:1.12.0

Reliable Transport

service Nexus Docker Downloads Portal Docker Helm Chart
Reliable Transport registry.dl.kx.com/kxi-rt:1.12.0 portal.dl.kx.com/kxi-rt:1.12.0 kxi-rt-1.12.0.tgz

Miscellaneous

service Nexus Docker package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.12.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

Deprecations

  • The RT Bridge has been deprecated as of the 1.12 release. The rt.qpk now offers full functional equivalency to the RT bridge so it is recommended to migrate to it prior to RT Bridge being removed.

Summary

We hope you find some useful features that optimize your kdb Insights SDK experience. Try them out and email our Support Team if you need any help.

We look forward to bring you even bigger features in kdb Insights SDK 1.13 coming soon!