kdb Insights Enterprise - Release Notes

This page details the release notes for the latest series of kdb Insights Enterprise releases. Key themes include improved access controls, usability and faster ingestion of real-time data into the database. To find the release notes for an older release, refer to the previous releases page.

1.7.3

Release Date

2023-11-01

Fixes

Expand to see the full detail of resolved issues here

Database

Optimizations made to the getData API to improve query performance when the prtnCol is sorted.

Security

Various CVEs have been remediated as part of this release.

Third-party dependencies

kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.7.3.tgz
Operator	kxi-operator-1.7.2.tgz
CLI	kxicli-1.7.2-py3-none-any.whl
C interface	kxi-c-sdk 1.7.0
Java interface	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.7.0.tgz

1.7.2

Release Date

2023-10-03

Improvements

SQL Queries
- SQL queries will now use less memory if late data is enabled and you are querying data for the current day.

Fixes

Expand to see the full detail of resolved issues here

Database

Resolved an issue where data for dates modified after the retention period was not getting moved from the first to the secondary HDB tier.
Improved the startup time for HDB DAPs, allowing for faster availability of processes that serve queries.

Stream Processor

The Apply Schema transform node now works with NULLs in string types.

Third-party dependencies

kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.7.2.tgz
Operator	kxi-operator-1.7.0.tgz
CLI	kxicli-1.7.1-py3-none-any.whl
C interface	kxi-c-sdk 1.7.0
Java interface	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.7.0.tgz

1.7.1

Release Date

2023-09-19

Fixes

Expand to see the full detail of resolved issues here

Database

Resolved an issue with the Storage Manager not rolling back aborted transactions if an EOD failed.
Deleting data from the HDB using a filter option now correctly performs the delete. Previously, this returned an error claiming that the column(s) being filtered didn’t exist.
EOIs are no longer delayed after an EOI recovery event. Previously, it was possible for EOIs to become held up if the process was terminated then subsequently recovered. The EOIs would still complete successfully, but with a delay.

Security

Resolved CVE-2023-37920 Certifi's removal of e-Tugra root certificate, by upgrading the spaCy dependency.

Stream Processor

Attempts to horizontally scale pipelines using the direct write feature will now explicitly error since this is not currently supported.

Deployments & Upgrades

Resolved the known issue in 1.7.0 in which a race condition could cause an upgrade to fail with an error when running kxi install upgrade.

Third-party dependencies

kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.7.1.tgz
Operator	kxi-operator-1.7.0.tgz
CLI	kxicli-1.7.1-py3-none-any.whl
C interface	kxi-c-sdk 1.7.0
Java interface	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.7.0.tgz

1.7.0

Release Date

2023-08-25

New Features

End-of-day writedown memory controls

The Storage Manager (SM) process now supports restricting memory consumption for data writedown, giving the opportunity for cost savings. Reducing the RAM allocated to SM may cause EOD to run longer than if RAM was unrestricted.

For guidance on tuning your RAM setup, see Performance best practices and the recommended system sizes for Azure here.
One-node RT

kdb Insights Enterprise now allows you to optionally deploy an RT Stream as a one-node cluster instead of a three-node cluster when deploying via the kdb Insights CLI.
- In a three-node cluster, RT provides fault tolerance and high availability (HA).
- In a one-node cluster, RT provides fault tolerance, but the cluster is not highly available. This configuration saves resources and reduces costs for workload use cases that do not need to be highly available. In the event that the single RT node goes down, the publisher then continues to write to its local RT log file. When the RT node restarts, the publisher retrieves any data that is present on the publisher node that it has not yet received.
See here for more details on RT architecture.
C Subscriber support in Reliable Transport

The C language interface for RT is now able to subscribe to the data on an RT stream. See here for more details.
Backfill a new column with derived data

When adding a new column to your schema, you can now define a custom function that will backfill the new column with derived data.
Publishing derived data from custom APIs

Custom APIs that generate derived data can now stream and/or persist their output.
Appending data on batch ingest

When ingesting data using the direct write feature, you now have the option to append data rather than replace it. See overwrite option for more details.
Data Entitlements for querying data Beta

When querying via REST or IPC, kdb Insights Enterprise now provides data entitlement restrictions for limiting user access to the data contained within specific deployed databases.
Parquet file reader Beta

Ingesting Parquet files from local storage or from an AWS S3 bucket using the q reader API, Python reader API and the UI is now enabled. This feature will be enabled for Azure and GCP and include a file watcher with file pattern matching in a future release.

Improvements

End-of-day writedown performance improvements

The Storage Manager (SM) process has been optimized to significantly improve its performance - up to 50-70% reduction in EOD duration. This improved EOD performance can reduce the memory allocated or the time taken for writedown, depending on your workloads and configuration.

For guidance and best practices on how to configure your schema and system to take advantage of these optimizations, review Performance best practices and the recommended system sizes for Azure here.
User interface
- The database schema editor has been completely redesigned with a new responsive table grid for entering column metadata details. The editor uses keyboard driven actions for efficient data entry. Backed by HTML canvas rendering, hundreds of columns can be added with ease.
- The Database configuration now supports different storage classes for the IDB and HDB (for example, FSx for Lustre). There is now a dropdown under Advanced Writedown settings for the File System that changes the mount storage class. See here for details.
- The code editors in the Pipeline Editor can now be used to evaluate q and Python code. See Executable Editors in the Building a Pipeline section for more information.
- Virtual queries are now available in Views. See Virtual data sources for more information.
- The ‘Database Wizard’ has been removed from the UI. Databases can now be created and configured using the unified database configuration screens.
- Query results in the UI are now shown using the timezone as returned by the query APIs for both Console and Table views. Previously, Table view would apply the user’s timezone which may have been different.
Readers
- When using cloud and file readers, you can now use filename pattern matching to ingest multiple files using a single reader. This expands the ability for you to build up ingest workloads that pull in multiple files. For example, using an S3 reader, you could read all CSV files in all folders within a bucket using the pattern:
```
s3://path/to/bucket/**/*.csv
```
- When using cloud and file readers you can now use a file watcher which is usually paired with filename pattern matching described above. The reader will continually watch for new files matching the file path(s) provided. See here for details on:
  - q APIs: .qsp.read.fromGoogleStorage, .qsp.read.fromAzureStorage and .qsp.read.fromAmazonS3.
  - Python APIs: .qsp.read.fromGoogleStorage, from_azure_storage and from_amazon_s3.
  - UI: Google Cloud Storage, Microsoft Azure Storage and Amazon S3.
  Not to be used with the Write Direct to HDB option
  
  The Watcher options should not be used when also choosing the Write Direct to HDB option in the Database Writer in the UI, the q or Python APIs.
- Advanced configuration options for the Kafka Reader node in the Pipeline Builder UI allow more flexible options for security protocol configuration or changes to fetch intervals, as seen when subscribing to an Azure Event Hub Kafka connector.
Reliable Transport
- Users can now launch KXI feeds from outside of their Kubernetes cluster and send data to their kdb Insights Enterprise deploy.
Logging
- RT Stream logs are now included in Database logs on the Diagnostics page, and they are also included in the Database Issues counts shown on the Database Overview page.
- When viewing a deployed pipeline, you can now click on an icon to jump straight to the pipeline logs.
Terraform Scripts
- The default node types for each cloud provider have been changed to ensure that the default node group uses memory-optimised instances, and the node group for rook-ceph has storage-optimised instances:
```
    AWS:

    default node pool: "r5.xlarge"

    rook-ceph node pool: "i3.2xlarge"

    Azure:

    default node pool: "Standard_E4_v3"

    rook-ceph node pool:”Standard_L8s_v2"

    GCP:

    default node pool: "n1-highmem-4"

    rook-ceph node pool:”n2-standard-8"
```
- The deployment of the nginx controller has been changed to be a Daemonset which will ensure consistent availability across all nodes.
- The standalone installation using the terraform client scripts now enables cloud native system metrics on GKE clusters.

Fixes

Expand to see the full detail of resolved issues here

Database - When using the Filter By Label option in the Query page, you can now use more than one label for the filter. In previous releases, additional labels beyond the first were not taken into account when the query was executed.

Fixed an issue where schema type conversions from strings to GUIDs would fail and cause a type error.
Fixed an issue on certain NFS-based system (namely Lustre) where the ‘current’ database pointer could get converted to a directory resulting in an unexpected error.

EISDIR: /data/idb/current.new -> /data/idb/current

An error has been fixed that would occur when updating an object storage tier that used an inventory file after a data migration.
Fixed an issue when querying with .kxi.getData using a groupBy operation on a foreign key column where the name of the foreign key column would be thrown and the query would fail to complete.
Fixed an issue where renaming columns in the schema UI would incorrectly delete the underlying column and create a new one with null data. Column renames in the UI now correctly preserve the column’s data.
A new warning has been added to the Resource Coordinator if no Aggregators connect to it. After a wait period, if an Aggregator continues not to connect, this will be escalated to an error to indicate that the system is not ready for queries.
Stream resources now correctly show the amount of resources to be used when an additional stream was added. Previously the stream resources were only showing the compute settings for a single stream instead of adding both together.
When the kdb Insights Database was configured with an object storage tier but an inventory file was not provided, the Storage Manager would error on startup with invalid object storage URI error. This has been changed to not attempt to create an inventory file if no configuration is explicitly provided.
Resolved an issue when attempting to migrate HDB data to Azure Blob Storage. Previously, this would fail with a signature error.
Fixed a potential race condition between NFS replication and DAP reload signals. It was previously possible that a DAP may reload before NFS replicated data which could result in data gaps. A new sequencing mechanism has been added to ensure DAPs do not reload until replication is complete.
Values in a symbol column will now be properly available immediately after a batch ingest. Previously, these would be missing until the next EOI.
Renaming a sort column now longer errors.

Stream Processor

q code in a pipeline’s Global Code field will now be executed with the same semantics as loading a .q file. This fixes issues around expressions incorrectly being concatenated, the handling of comments, and expressions starting with .
The return of UDFs when generating q objects is no longer incorrectly wrapped as foreign objects. A user no longer needs to use a null python function following the UDF to unwrap it to the correct type.

Command Line Interface - Reinstalls and rollbacks no longer fail due to password history issues with the initUser.

Third-party dependencies

kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.7.0.tgz
Operator	kxi-operator-1.7.0.tgz
CLI	kxicli-1.7.0-py3-none-any.whl
C interface	kxi-c-sdk 1.7.0
Java interface	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.7.0.tgz

Upgrade notes

In 1.7.0, upgrades could occasionally fail with an error when running kxi install upgrade. This occurs due to a race condition of the CLI when trying to replace the CRDs. The upgrade will stall and timeout at this point.
```
..
Replacing CRD assemblies.insights.kx.com
Replacing CRD assemblyresources.insights.kx.com
Error: Timed out waiting for CRD assemblyresources.insights.kx.com to be deleted
```
Re-running the upgrade should workaround the issue. This is resolved in the 1.7.1 patch release.

Known Issues

With entitlements enabled, the Query tab in the UI will not return data as there are issues with resolving the correct database name when checking entitlements. In order to workaround this issue, you should create a second entitlement for each database / assembly.

For example if you create and entitle an assembly called dfx-assembly, you should also create an identical entitlement called dfx-assembly-qe. This -qe entitlement should be kept in-sync when updating the main one.

This restriction will be removed in a future release.
Previously cloud readers in a failure recovery scenario would begin reading file(s) from the start, even if they had read them prior to crashing. This would have caused duplicate data if using a stream or database writer. As of 1.7 this is no longer the case. One exception case remains, when using a glob pattern match in a non watching scenario. If the pipeline fails, recovers and matches a new set of files (because the underlying files in the object storage have changed) we may get duplicate data.
qsql requests, made either through the Q tab (query environment) in the UI or via REST to /qsql, do not work with entitlements. The .kxi.qsql API should be disabled if using entitlements.

Backward Compatibility

If you are upgrading from a version earlier than 1.6.3 you must also read each of the relevant release notes to ensure you know all the changes that will affect the upgrade. Refer to the previous release page for older releases.