Skip to content

kdb Insights Microservices - Release Notes

This page details the release notes for the latest series of kdb Insights Microservices releases. To find the release notes for an older release, refer to the previous releases page.

1.10.0

Release Date 2024-06-20

The main focus of this kdb Insights 1.10 release is a set of incremental updates designed to simplify system administration and streamline user experience.

New Features

  1. SQL2 now supports joins
  2. kxi-python now supports publish, query and execution of custom APIs
  3. Stream Processor supports subscribing to the end of an RT stream
  4. Natural language descriptions included in getMeta descriptions
  5. Cost Reduction Options for Reliable Transport

1. SQL2 now supports joins

Joins support in SQL2 enables you to combine multiple tables and/or dictionaries in your workflows using either INNER, LEFT, RIGHT, FULL, or CROSS joins. To find out more about streamlining your data connectivity across datasets, refer to the documentation.

2. kxi-python now supports publish, query and execution of custom APIs

The Python interface to kdb Insights Enterprise, kxi-python, has been extended to allow for publishing to your kdb Insights Enterprise deployment. Refer to the documentation as a guide for how to both publish and query data using it. kxi-python also now supports the execution of custom APIs against your deployment, as shown in this example. These enhancements promote a stronger developer experience by significantly improving efficiency and streamlining workflows.

3. Stream Processor supports subscribing to the end of an RT stream

This release introduces the ability to subscribe to the end of an RT Stream. When configuring the kdb Insights Stream reader, it now provides an end parameter which will subscribe to the latest position in the stream instead of the beginning. This enables late joiners to reduce consumption, thus enhancing operational efficiency. Refer to the documentation links to learn more:

4. Natural language descriptions included in getMeta descriptions

getMeta descriptions now include natural language descriptions of tables and columns in its result, enabling you to attach and retrieve detailed descriptions of database structures. Read more in our documentation.

5. Cost Reduction Options for Reliable Transport

An alternative topology for a publisher to choose when sending data to RT has been added. This alternative has the advantage of reducing the ingress bandwidth to a third of what the default topology requires. This could offer significant cost savings where ingress data incurs charges. Refer to the documentation for details.

Improvements

Performance Improvements in getData

For queries that span multiple partitions, getData has been optimized to improve the query execution times.

End-of-Interval Memory Optimization

End-of-Interval (EOI) Memory Optimization ensures that large, splayed tables are automatically cleared at EOI. This enhances memory efficiency and reduces RAM dependency, resulting in accelerated system performance and smoother system operations. There is no user configuration required to take advantage of this enhancement. See the documentation for information on splayed tables.

getData REST Request Filtering

REST requests in getData now support filtering on time, minute, and timespan data type fields.

Streaming JSON responses over HTTP in the Service Gateway

The Service Gateway (SG) now supports returning JSON responses in a streaming fashion without the need to compress the result. This helps reduce the necessary amount of RAM the SG needs in order to handle larger payloads. The environment variable KXI_SG_STREAM_THRESHOLD will allow you to configure your chosen chunk size for splitting up the overall response. This capability is automatic in the Service Gateway and does not require any user configuration. See the documentation here for more details.

Stream Processor Kafka Writer

The SP Kafka Writer now supports an options parameter for specifying advanced configuration on the Kafka broker connection. Previously, it only supported advanced configuration on the topic via the topicConfig parameter. See the documentation for information.

Fixes

Expand to see the full detail of resolved issues here

Database

  • For a new system with an empty database, queries would not work after the first end-of-day data writedown if no data had been ingested yet. This issue has been resolved.

  • Previously, a user would need to manually edit their par.txt file if the object storage location information was changed. This has been resolved so that par.txt is updated automatically after completing the required restart.

  • When using nested filters in getData on keyed partitioned tables with late data, updates to on-disk data were not being reflected in the output of getData if the data had not yet been written to disk.

  • Importing a database from object storage is now correctly sourcing the database's sym file upon initialization of the Storage Manager.

  • Fixed an issue whereby getData REST queries would produce inf instead of a null. This inf response would break most JSON parsers.

  • Prior to 1.10.0, the Service Gateway was configuring the minimum heap size to retain equal to the container requests. This could result in the container holding onto unused memory and not releasing it back to the OS. Additionally, the gateway was not setting an explicit size for off-heap memory used by IO messages, which could result in using a combined total of heap and off heap memory exceeding the container limit.

    To avoid both of these issues, set JAVA_OPTS explicitly. Use -Xms to set the minimum heap size to always retain, -Xmx to set the maximum heap size, and -XX:MaxDirectMemorySize to set the maximum size used for IO messages off heap. We recommending splitting the container limits equally between heap and IO.

    Example configuration of 256Mb minimum reserved heap and 2G limit, split into 1g for heap and direct memory:

    `JAVA_OPTS="-Xms256M -Xms1G -XX:MaxDirectMemorySize=1G"`
    

Reliable Transport

  • Previously, the RT client could fail to recognize a new log file, forcing downstream subscribers into a hung state. This has been resolved.

Cloud Readers

  • A file watcher in Azure reader did not work for test or full deploys.

Kubernetes Discovery

  • Previously, the KXI_RC_LABEL_SELECTOR field was not being correctly honored, causing Kubernetes discovery to become non-functional.

Security

  • Various CVEs have been remediated as part of this release.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

Stream Processor

microservice Nexus Docker Downloads Portal Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.10.0 portal.dl.kx.com/kxi-sp-controller:1.10.0
Worker registry.dl.kx.com/kxi-sp-worker:1.10.0 portal.dl.kx.com/kxi-sp-worker:1.10.0
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.10.0 portal.dl.kx.com/kxi-sp-coordinator:1.10.0
Worker-Python registry.dl.kx.com/kxi-sp-python:1.10.0 portal.dl.kx.com/kxi-sp-python:1.10.0
Worker-ML registry.dl.kx.com/kxi-ml:1.10.0 portal.dl.kx.com/kxi-ml:1.10.0

Database

microservice Nexus Docker Downloads Portal Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.10.0 portal.dl.kx.com/kxi-da:1.10.0
Single container registry.dl.kx.com/kxi-da-single:1.10.0 portal.dl.kx.com/kxi-da-single:1.10.0
Storage Manager registry.dl.kx.com/kxi-sm-single:1.10.0 portal.dl.kx.com/kxi-sm-single:1.10.0
Gateway registry.dl.kx.com/kxi-sg-gw:1.10.0 portal.dl.kx.com/kxi-sg-gw:1.10.0
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.10.0 portal.dl.kx.com/kxi-sg-rc:1.10.0
Aggregator registry.dl.kx.com/kxi-sg-agg:1.10.0 portal.dl.kx.com/kxi-sg-agg:1.10.0

Reliable Transport

microservice Nexus Docker Downloads Portal Docker Helm Chart
Reliable Transport registry.dl.kx.com/kxi-rt:1.10.0 portal.dl.kx.com/kxi-rt:1.10.0 kxi-rt-1.10.0.tgz

Miscellaneous

microservice Nexus Docker package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.10.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar