Skip to content

kdb Insights SDK - Release Notes

This page details the release notes for the latest series of kdb Insights SDK releases. To find the release notes for an older release, refer to the previous releases page.

1.11.0

Release Date 2024-09-20

kdb Insights SDK 1.11 brings increased security measures with data-at-rest encryption, new query features in the getData API, greater configuration options, and enhancements to stability and performance. And so much more! See below for the full details.

New Features

  1. Data At Rest Encryption (DARE)
  2. Reliable Transport's (RT) client replica size discovery
  3. Improvements to user-defined analytics
  4. LIMIT support in getData

1. Data At Rest Encryption (DARE)

Encrypt all tables in your database using kdb+'s native DARE capability. When enabled, all on-disk data will be encrypted using AES256CBC. DARE enables customers to comply with their data security requirements. Read the encryption at rest section for details.

2. Reliable Transport's (RT) Client Replica Size Discovery

Enhancements to RT simplify client connection and configuration setup. This includes automatic discovery of the cluster size, eliminating the need for clients to specify sizing manually and allowing for seamless resizing of the RT cluster. This change enhances the user experience by reducing the configuration needed and streamlining setup across all interfaces (C, q, Python, and Java). Explore the enhancements in Getting started.

3. Improvements to User Defined Analytics (UDAs)

We have enhanced the documentation and consolidated User Defined Analytics (UDAs) development into a single, unified API. This update simplifies the process for developers by eliminating the need for detailed internal system knowledge and streamlines API functionality across various components. This release also introduces the ability for a UDA to call another UDA. Read the UDA introduction for an overview of UDAs and details on how to use the new functions. In addition, please read the Deprecations section below, which outlines name changes to important functions.

4. limit support in getData

The getData API now supports a limit clause, allowing you to retrieve a subset of data (for example, the top 100 rows). This enhancement increases the reliability of queries on large datasets and offers more efficient data handling. It improves the user experience when running large queries, reducing the risk of the query failing. Read limit for details on how to use limit in getData. Refer to row limit for guidance on setting limits in the Query builder. You can also take advantage of this when using Views.

Improvements

Performance improvements for Storage Manager operations

  • Optimizations in the Storage Manager have been made to improve the efficiency of schema conversions, including the introduction of multi-threading. This significantly reductes the time taken to perform a schema change operation. Read multi-threading for more details.

  • Reduction in the RAM required for re-enumeration when carrying out a batch ingest.

  • Adding/removing an object storage tier to existing databases is now supported. Read here for details.

SQL

We have introduced support in SQL2 for the following:

  • a new keyword OFFSET

  • multi-directional ORDER BY. An example is seen below.

    select * from trade order by sym asc, price desc

getData API

The following improvements have been introduced in 1.11:

  • support for descending and mixed sorting in getData has been added.
  • getData now supports OFFSET as part of its limit implementation. Read getData limits for details.
  • getData now validates that functions are called with the correct number of parameters.
  • the ability to bind view state variables easily into filter statements in getData. Refer to Using view states for details.

File Watchers

  • The local file reader now supports file watching. See the documentation for q and Python for details.

Reliable Transport (RT)

  • The kxi-rt helm chart for RT will no longer provision load balancers by default.

  • The RT Java interface now allows clients to connect to RT and provide the configuration from code. See here for details.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Fixed an issue where getData queries failed with a type error if the time range of the request landed completely between two IDB partitions.

  • Fixed an issue where getData was not correctly enlisting the target symbol when using <> like it does when using =.

  • getData now validates that functions are called with the correct number of parameters.

  • getData filters of the form ["like", "col", x] where col is a string/symbol column and x is a string, symbol, or char, now work without throwing a type error.

  • Fixed an issue where getData REST queries would produce inf instead of a null.

  • Batch ingest direct write pipelines now correctly honor compression and encryption settings of an HDB tier.

Readers

  • Resolved an issue where the Database reader was throwing type errors when filters were applied. These queries now function correctly, including in the Scratchpad UI.

Security

  • Various CVEs have been remediated as part of this release.

Deprecations

  • As a result of the improvements to how user-defined analytics work, the following functions are being deprecated but will still be available to use:

    • .kxi.selectTable has gone from a 6-parameter function to a single function that takes a single dictionary argument.
    • .da.getTableMem replaced by .kxi.getTableBuffer
    • .da.getTableDelta replaced by .kxi.getTableOverflow
    • .da.i.dapType (variable) replaced by .kxi.getDapType function
    • .sapi.{ok,hok} replaced by .kxi.response.{ok,hok}
    • .sapi.response replaced by .kxi.response.make
    • .sapi.defer replaced by .kxi.response.callAPI
    • DAP UDAs must return a response with one of the .kxi.response.{*} functions, e.g. .kxi.response.ok
    • .da.registerAPI and .sgagg.registerAggFn replaced by .kxi.registerUDA
    • .sapi.meta{Param,Return,Description,Misc} replaced by .kxi.meta{Param,Return,Description,Misc}
    • .kxi.getMeta v2 replaced by v3.
    • KXI_SAPI_HB_{FREQ,TOL} replaced by KXI_HB_{FREQ,TOL}

  • When configuring a process (Aggregator, DAP, RC) , the environment variable KXI_SAPI_HB_FREQ set the time in milliseconds to run the heartbeat to connected processes, and KXI_SAPI_HB_TOL to set the number of heartbeat intervals a process can miss before being disconnected (default: 2). These variables have been updated to remove the deprecated reference to SAPI (see above). These configurations are now KXI_HB_FREQ and KXI_HB_TOL respectively. The old values will be supported but are deprecated, and will be sunsetted in future.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

Stream Processor

service Nexus Docker Downloads Portal Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.11.0 portal.dl.kx.com/kxi-sp-controller:1.11.0
Worker registry.dl.kx.com/kxi-sp-worker:1.11.0 portal.dl.kx.com/kxi-sp-worker:1.11.0
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.11.0 portal.dl.kx.com/kxi-sp-coordinator:1.11.0
Worker-Python registry.dl.kx.com/kxi-sp-python:1.11.0 portal.dl.kx.com/kxi-sp-python:1.11.0
Worker-ML registry.dl.kx.com/kxi-ml:1.11.0 portal.dl.kx.com/kxi-ml:1.11.0

Database

service Nexus Docker Downloads Portal Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.11.0 portal.dl.kx.com/kxi-da:1.11.0
Single container registry.dl.kx.com/kxi-da-single:1.11.0 portal.dl.kx.com/kxi-da-single:1.11.0
Storage Manager registry.dl.kx.com/kxi-sm-single:1.11.0 portal.dl.kx.com/kxi-sm-single:1.11.0
Gateway registry.dl.kx.com/kxi-sg-gw:1.11.0 portal.dl.kx.com/kxi-sg-gw:1.11.0
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.11.0 portal.dl.kx.com/kxi-sg-rc:1.11.0
Aggregator registry.dl.kx.com/kxi-sg-agg:1.11.0 portal.dl.kx.com/kxi-sg-agg:1.11.0
Rollback registry.dl.kx.com/kxi-db-rollback:1.11.0 portal.dl.kx.com/kxi-db-rollback:1.11.0

Reliable Transport

service Nexus Docker Downloads Portal Docker Helm Chart
Reliable Transport registry.dl.kx.com/kxi-rt:1.11.0 portal.dl.kx.com/kxi-rt:1.11.0 kxi-rt-1.11.0.tgz

Miscellaneous

service Nexus Docker package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.11.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

Summary

We hope you find some useful features that optimize your kdb Insights SDK experience. Try them out and email our Support Team if you need any help.

We look forward to bring you even bigger features in kdb Insights SDK 1.12 coming soon!