Skip to content

kdb Insights Enterprise - Release Notes

This page details the release notes for the latest series of kdb Insights Enterprise releases. To find the release notes for an older release, refer to the previous releases page.

1.10.1

Release Date 2024-07-24

Fixes

Expand to see the full detail of resolved issues here

Packaging

  • Resolved an issue in version 1.10.0 where, when removing a pipeline from a package using the UI, the pipeline would revert to a default pipeline specification instead of being removed.

  • Resolved an issue where adding a pipeline into two databases caused the UI to not behave correctly.

Database

  • Fixed an issue where getData queries would fail with a type error if the time range of the request landed completely between two IDB partitions.

  • An embedded assembly Resource Coordinator could become unreachable if its connection with the Service Gateway dropped.

  • Batch ingest has been modified to reduce the amount of RAM required for re-enumeration and sorting during a batch ingest operation.

  • Fixed an issue where batch ingest in directWrite mode did not properly apply primaryKeys to the data. Prior to this release, the data would not be properly collapsed, resulting in multiple rows for each primary key being persisted to disk. See the following example:

The data was ingested using batch ingest with `directWrite` mode enabled into table `t` that has primary keys configured as `pk1`, `pk2`

pk1 pk2 val
-----------
a   1   0
a   1   1
b   1   2
b   2   3
b   1   4
c   4   5

Prior to this release, the below data would be persisted to disk in the HDB:

pk1 pk2 val
-----------
a   1   0
a   1   1
b   1   2
b   2   3
b   1   4
c   4   5

The behaviour has been fixed to correctly apply the primary keys to the data, meaning that the below rows would now be persisted to the HDB:

pk1 pk2 val
-----------
a   1   1
b   1   4
b   2   3
c   4   5

If tables are configured with primary keys, then after upgrading to 1.10.1, you may notice a decrease in the number of records being persisted when compared to data in older partitions. This is due to the corrected behavior.

If you have data from a prior release that was ingested via batch ingest with directWrite mode enabled, it is possible to correct it. For assistance with this, contact KX Support.

Readers

Pipelines

  • In version 1.10.0, pipelines with an Apply node(s) from releases older than version 1.9 would fail on deployment. This has been resolved, and older pipelines that use Apply nodes now work as expected without any user action required.

Third-party Dependencies

kdb Insights Enterprise 1.10.1 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.10.0 release of the standalone infrastructure installation scripts.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type Nexus location Downloads Portal location
Enterprise insights-1.10.1.tgz insights-1.10.1.tgz
Operator kxi-operator-1.10.1.tgz kxi-operator-1.10.1.tgz
CLI kxicli-1.10.1-py3-none-any.whl kxicli-1.10.1-py3-none-any.whl
RT C interface kxi-c-sdk 1.10.0 kxi-c-sdk 1.10.0
RT Python interface kxi-rtpy-1.10.0 kxi-rtpy-1.10.0
RT Java interface kxi-java-sdk 1.10.0 kxi-java-sdk 1.10.0
RT Bridge rt-bridge 1.7.0 rt-bridge 1.7.0
Infrastructure kxi-terraform-1.10.1.tgz kxi-terraform-1.10.1.tgz
kxi-management-service kxi-management-service-0.2.4.tgz kxi-management-service-0.2.4.tgz
insights-on-k8s insights-on-k8s-0.1.6.tgz insights-on-k8s-0.1.6.tgz

1.10.0

Release Date 2024-06-20

The main focus of this kdb Insights 1.10 release is a set of incremental updates designed to simplify system administration and streamline user experience.

New Features

  1. SQL2 now supports joins
  2. Standardized Auditing
  3. Environment Variables in Packages
  4. kxi-python now supports publish, query and execution of custom APIs
  5. Publishing to Reliable Transport (RT) using the CLI
  6. Stream Processor supports subscribing to the end of an RT stream
  7. Schema Conversion Progress Indicator
  8. Natural language descriptions included in getMeta descriptions
  9. Cost Reduction Options for Reliable Transport

1. SQL2 now supports joins

Joins support in SQL2 enables you to combine multiple tables and/or dictionaries in your workflows using either INNER, LEFT, RIGHT, FULL, or CROSS joins. To find out more about streamlining your data connectivity across datasets, refer to the joins documentation.

2. Standardized Auditing

Standardized auditing provides reliable and consistent event logging, bolstering system security, and event accountability. The new structured framework introduced in this release adapts to current and future auditing needs aligned with specific services, to ensure comprehensive monitoring and robust accountability. Find out more about the audit framework in the Logging documentation.

3. Environment Variables in Packages

Environment Variables in Packages enables administrators to inject environment variables at runtime and dynamically pass values to both the database and pipelines. Variables can be set globally or per component and are applicable for custom analytics through global settings. Learn how to implement environment variables in the Package deployment documentation.

4. kxi-python now supports publish, query and execution of custom APIs

The Python interface to kdb Insights Enterprise, kxi-python, has been extended to allow for publishing to your kdb Insights Enterprise deployment. Refer to the documentation as a guide for how to both publish and query data using it. kxi-python also now supports the execution of custom APIs against your deployment, as shown in this example. These enhancements promote a stronger developer experience by significantly improving efficiency and streamlining workflows.

5. Publishing to Reliable Transport (RT) using the CLI

You can now use KXI CLI to publish ad-hoc messages to kdb Insights Enterprise using Reliable Transport. This change empowers you to use tools such as VS Code to publish data directly to the database, simplifying integration and real-time insights. This documentation demonstrates how to publish data using the CLI to kdb Insights Enterprise.

6. Stream Processor supports subscribing to the end of an RT stream

This release introduces the ability to subscribe to the end of an RT Stream. When configuring the kdb Insights Stream reader, it now provides an end parameter which will subscribe to the latest position in the stream instead of the beginning. This enables late joiners to reduce consumption and enhancing operational efficiency. Refer to the documentation links to learn more:

7. Schema Conversion Progress Indicator

A new schema conversion progress indicator provides data engineers and developers with visibility of the schema conversion process, including progress and error reporting.

This is a useful tool for monitoring the progress of schema conversions, especially for large data sets which require a significant amount of time.

8. Natural language descriptions included in getMeta descriptions

getMeta descriptions now include natural language descriptions of tables and columns in its result, enabling you to attach and retrieve detailed descriptions of database structures. Read more in our documentation.

9. Cost Reduction Options for Reliable Transport

An alternative topology for a publisher to choose when sending data to RT has been added. This alternative has the advantage of reducing the ingress bandwidth to a third of what the default topology requires. This could offer significant cost savings where ingress data incurs charges. Refer to the documentation for details.

Improvements

Performance Improvements in getData

For queries that span multiple partitions, getData has been optimized to improve the query execution times.

End-of-Interval Memory Optimization

End-of-Interval (EOI) Memory Optimization ensures that large, splayed tables are automatically cleared at EOI. This enhances memory efficiency and reduces RAM dependency, resulting in accelerated system performance and smoother system operations. There is no user configuration required to take advantage of this enhancement. See the documentation for information on splayed tables.

Updated Entitlement Administration

The commands to use entitlements with the CLI have changed. Refer to Configuring data entitlements for details.

getData REST Request Filtering

REST requests in getData now support filtering on time, minute, and timespan data type fields.

Streaming JSON responses over HTTP in the Service Gateway

The Service Gateway (SG) now supports returning JSON responses in a streaming fashion without the need to compress the result. This helps reduce the necessary amount of RAM the SG needs in order to handle larger payloads. The environment variable KXI_SG_STREAM_THRESHOLD will allow you to configure your chosen chunk size for splitting up the overall response. This capability is automatic in the Service Gateway and does not require any user configuration. See the documentation here for more details.

Customizable Polling Frequency in File Watchers

Cloud-polling file watchers now support the ability to configure the frequency in which polling for new files is done. When configuring a pipeline, the Use Watching checkbox provides a Timer option for selecting a frequency to poll by. See the documentation describing the optional parameter.

Stream Processor Kafka Writer

The SP Kafka Writer now supports an options parameter for specifying advanced configuration on the Kafka broker connection. Previously, it only supported advanced configuration on the topic via the topicConfig parameter. See the documentation for information.

Filtering Rows in Views

Data sources in Views now support the Max Rows option that limits the maximum number values displayed.

Select Columns

In the UI Query screen, it is now possible to filter a query by a selected column(s) to refine the data returned by the query. This allows users to reduce payload sizes and avoid potentially running out of RAM.

Streaming to Views

Streaming data sources now include a Max Rows option which facilitates the use of a rolling window chart that shows the last x number of points.

The Streaming to Views walkthrough has been updated with a new section Filtering based on a dropdown to show how to filter a Data Grid to a specific value from a list.

Fixes

Expand to see the full detail of resolved issues here

Database

  • For a new system with an empty database, queries would not work after the first end-of-day data writedown if no data had been ingested yet. This issue has been resolved.

  • Previously, a user would need to manually edit their par.txt file if the object storage location information was changed. This has been resolved so that par.txt is updated automatically after completing the required restart.

  • The UI was erroneously reporting that a schema change was in progress in cases where an error in conversion had occurred. This issue has been resolved.

  • When using nested filters in getData on keyed partitioned tables with late data, updates to on-disk data were not being reflected in the output of getData, if the data had not yet been written to disk. This issue has been resolved.

  • Importing a database from object storage is now correctly sourcing the database's sym file upon initialization of the Storage Manager.

  • Previously, getData REST queries would produce inf instead of a null. This inf response would break most JSON parsers. This issue has been resolved.

  • Prior to 1.10.0, the Service Gateway was configuring the minimum heap size to retain equal to the container requests. This could result in the container holding onto unused memory and not releasing it back to the OS. Additionally, the gateway was not setting an explicit size for off-heap memory used by IO messages, which could result in using a combined total of heap and off heap memory exceeding the container limit.

    To avoid both of these issues, set JAVA_OPTS explicitly. Use -Xms to set the minimum heap size to always retain, -Xmx to set the maximum heap size, and -XX:MaxDirectMemorySize to set the maximum size used for IO messages off heap. We recommend splitting the container limits equally between heap and IO.

    Example configuration of 256Mb minimum reserved heap and 2G limit, split into 1g for heap and direct memory: 

    `JAVA_OPTS="-Xms256M -Xms1G -XX:MaxDirectMemorySize=1G"`

Reliable Transport

  • Previously, the RT client could fail to recognize a new log file, forcing downstream subscribers into a hung state. This has been resolved.

UI

  • On the Resource tab, incorrect database resource summary values were displayed when no limits were specified. Now, if the limit value is empty, the request value is used to calculate summary values.

  • Previously, the /details API didn't always show pipelines that were still in a TEARING_DOWN status.

  • When configuring a pipeline in the UI, setting additional workerThreads did not correctly apply the specified configuration. However, setting the environment variable KXI_DEFAULT_SP_WORKER_THREADS did work. This has been resolved so either method can be used.

Cloud Readers

  • A file watcher in Azure reader did not work for test or full deploys.

  • An issue has been resolved where, when administering a pipeline, a cloud reader node (s3, ms, and gs) could prompt you that there were unsaved changes even when no changes have been made.

Packages

  • Package names created in the CLI are now restricted to only use lower case letters and limited to 24 characters, inline with the GUI and Kubernetes restrictions. Previously, the deployment would fail because Kubernetes was unable to deploy an assembly with capital letters in its name.

  • Previously, exporting a pipeline did not complete successfully. This is now fixed.

Kubernetes Discovery

  • Previously, the KXI_RC_LABEL_SELECTOR field was not being correctly honored, causing Kubernetes discovery to become non-functional.

Scratchpads

  • Using a file watcher (ex. await pending file) was not working in a Scratchpad session.

Security

  • Various CVEs have been remediated as part of this release.

Third-party Dependencies

kdb Insights Enterprise 1.10.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.10.0 release of the standalone infrastructure installation scripts.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type Nexus location Downloads Portal location
Enterprise insights-1.10.0.tgz insights-1.10.0.tgz
Operator kxi-operator-1.10.0.tgz kxi-operator-1.10.0.tgz
CLI kxicli-1.10.0-py3-none-any.whl kxicli-1.10.0-py3-none-any.whl
RT C interface kxi-c-sdk 1.10.0 kxi-c-sdk 1.10.0
RT Python interface kxi-rtpy-1.10.0 kxi-rtpy-1.10.0
RT Java interface kxi-java-sdk 1.10.0 kxi-java-sdk 1.10.0
RT Bridge rt-bridge 1.7.0 rt-bridge 1.7.0
Infrastructure kxi-terraform-1.10.0.tgz kxi-terraform-1.10.0.tgz
kxi-management-service kxi-management-service-0.2.3.tgz kxi-management-service-0.2.3.tgz
insights-on-k8s insights-on-k8s-0.1.6.tgz insights-on-k8s-0.1.6.tgz

Upgrade Notes

New Requirements for Hostnames & kxi package

When you run a kxi package command, the kxi cli first authenticates against kdb Insights Enterprise. In 1.9, running a kxi package command would look up the cached hostname that had been previously authenticated against in the ~/.insights/credentials file. In 1.10.0, the hostname previously authenticated against is no longer used. Instead, you must set the hostname in your ~/.insights/cli-config.

Shutting Down Databases Before Upgrading

The operational procedure described below is no longer required for an upgrade to 1.10.0. However, it is still true for upgrading from 1.8 to 1.9.

  • Before you upgrade to 1.9.x, if you have any running databases, it is advised to manually stop them before upgrading, otherwise you may become unable to edit them in the UI. If this happens, you can recover the editable state by considering the following options:

    • If possible, back up the running assembly yaml file with kubectl get asm
    • Tear down the running assembly manually with kubectl delete asm
    • Optionally, use the kxi package deploy command in the CLI, specifying the name and version of the package. The package appears under kxi package remote-list. Refer to Package deployment documentation for more information.

    If the package doesn't reappear as editable after tearing down, you need to use the KXI CLI to perform the following operations:

Known Issues

Expand to see the list of known issues here

Resiliency With Resource Coordinators Deployed via kxi package

  • If the Resource Coordinator (RC) is deployed using kxi package, and a connection is lost between the Service Gateway (SG) and the RC, queries routed to said RC will stop being serviced. This does not occur in cases where the deployment was done with the base Insights Enterprise install.

If the connection is lost and queries are failing, this can be resolved by restarting the Service Gateway.

PostGreSQL & Keycloak Resiliency

  • A transient issue can occur in which Keycloak is unable to re-establish a connection to PostgreSQL after it recovers from an outage. This can result in getData being unable to serve queries. To work around this, restart PostgreSQL.

Pipelines

  • Pipelines that use Apply nodes from releases older than 1.9 will fail on deployment after upgrading. To work around this, after performing your upgrade, open the pipeline, click on Apply node, click the gear icon in the top corner of the node-config area, and then set the parameters manually (for most apply nodes this will be operator, metadata, data).

  • Deploying a pipeline from the UI will mark it as having unsaved changes even when none exist. Clicking Save again will make this go away.

Upgrades & Rollbacks

  • Users are prompted to tear down assemblies upon rollback. The rollback should exit if the user responds ‘n' to this prompt. Instead, the rollback continues.

  • During upgrade, the postgresql process may log errors similar to the below. These can be safely ignored.

    `FATAL: database `[database name]` does not exist`

UI

Packaging

  • In the UI, when you remove a pipeline from a database, packaging preserves this pipeline in a separate package to ensure it is not permanently deleted. This means the pipeline is still available in the UI. However, if you re-push the package from the CLI that still contains the pipeline, the pipeline now exists in two packages, and the latest version of the pipeline is not being served to the UI. The workaround is to remove the pipeline-only package that has been created.

Beta Features

Summary

We hope you find some useful features that optimize your kdb Insights Enterprise experience. Try them out and email our Support Team if you need any help.

We look forward to bring you even bigger features in kdb Insights Enterprise 1.11 coming soon!