kdb Insights Enterprise - Release Notes

This page details the release notes for the latest series of kdb Insights Enterprise releases. To find the release notes for an older release, refer to the previous releases page.

1.5.2

Release Date

2023-05-31

Improvements

In the Query Window of the kdb Insights Enterprise UI, distinct aggregation is now available as a filter option.
The getData API can now perform aggregations on foreign key columns.

Fixes

Expand to see the full detail of resolved issues here

Infrastructure

Kubernetes service account tokens are now automatically refreshed from disk. In previous versions these were not refreshed, causing failing requests to the Kubernetes control plane and a need to restart your pods to recover.

Database

When deploying an assembly via the kdb Insights CLI, if size is not specified for DAP instances, a default of 3 replicas will once again be applied. In 1.5.1, this default was increased to 4 to provide increased availability, but 3 has been restored to reduce costs. Users can configure a greater number of replicas if desired.
In the kdb Insights Enterprise UI, pipelines associated with a database now show in the status table. Previously, only the first pipeline would be shown.
In the kdb Insights Enterprise UI, the database schema overview now updates when changes are made to the database’s schema. Previously, this would only show the schema state when first viewed.
In the kdb Insights Enterprise UI, database documents no longer get marked as modified after a teardown or cleanup.
The database configuration UI now validates that the minimum rollover time is 1 minute. Previously, this could be set to 0.

Stream Processor

Pipelines can be now be safely torn down without checkpoints having been cleared. Previously, if a teardown was followed by a restart, it is possible the pipeline would never run.

Artifacts

type	location
Enterprise	insights-1.5.2.tgz
Operator	kxi-operator-1.5.2.tgz
CLI	kxicli-1.5.0-py3-none-any.whl
C SDK	kxi-c-sdk 1.5.0
Java SDK	kxi-java-sdk 1.5.0
RT Bridge	rt-bridge 1.5.0
Infrastructure	kxi-terraform-1.5.1.tgz

1.5.1

Release Date

2023-05-19

Improvements

SQL queries that filter by time but do not explicitly select the time column are now supported.
Query timeout messages now indicate a reason for the timeout. Previously, query timeouts would give no reason or say “timed out for unspecified reasons”. See queueing timeouts for more details.

Fixes

Expand to see the full detail of resolved issues here

Database

Configuring a stream from the UI is now correctly applying the specified stream size for RT log files. This was previously being defaulted to 20Gi.
Clicking ‘Save & Deploy’ on the database wizard with a default schema no longer errors. Previously, if the schema on the database wizard was not modified from the default, it would error with an invalid sort column.
Databases without object storage tiers will no longer error and restart to resume normal operations after a schema conversion is performed.
New validation has been added to ensure that the snap value set in the EOD schedule of an assembly is always a time value. Previously, some examples showed usage of this field with a timestamp which was incorrectly ignored.
To improve resiliency, the default number of DAPs deployed for an assembly is now 4. Previously this was 3 and did not guarantee queries could complete if one of the DAPs was offline.
New validation has been added to the kdb Insights Enterprise UI to ensure that a database always contains a partitioned table. Additional validation has been added to ensure that the Parted attribute is only set on one column-tier combination within a table. For example, setting Parted for a temporal on-disk attribute can only be set on one column, but a different column could have Parted set on the ordinal on-disk attribute.
Sidecar containers are no longer automatically registering to object storage IAM service accounts where available. With large deployments, this could result in causing authentication backoffs for more mission critical processes.
In the kdb Insights Enterprise UI, on the Database settings page, the ‘Total Query Resources’ now accounts for the ‘Exploratory Read Replicas’ count in its calculation. Previously this value was not accounting for these replicas and was underestimating the resources allocated. <!-https://kxl.atlassian.net/browse/KXI-22551→
The kxi_da_request_time metric has been changed to use a histogram type to reduce metric output. This was previously a summary type.
A new connection heartbeat has been added from DAPs to RC to improve connection stability when an RC restarts and is scheduled with a new IP.
Enabling an external stream before saving a database in the kdb Insights Enterprise UI is now supported. Previously, enabling an external stream before saving would make the stream disabled after saving.

User Interface - When renaming a pipeline from the navigation sidebar, the new name is now properly displayed in the Save and Deploy dialogs.

Corrected the behaviour causing a 'Timed out while updating resources' message to be displayed when Deploy is attempted shortly after a Teardown, or the other way around.

Artifacts

type	location
Enterprise	insights-1.5.1.tgz
Operator	kxi-operator-1.5.1.tgz
CLI	kxicli-1.5.0-py3-none-any.whl
C SDK	kxi-c-sdk 1.5.0
Java SDK	kxi-java-sdk 1.5.0
RT Bridge	rt-bridge 1.5.0
Infrastructure	kxi-terraform-1.5.1.tgz

1.5.0

Release Date

2023-05-09

New Features

UI

The kdb Insights Enterprise UI has a new and improved look and user experience for viewing the details of your database(s). All relevant details about a database are now grouped together in a single summary page, giving users an intuitive view into the details and health of its components. Users can quickly see if any errors or warnings are present, and drill down into the details to investigate directly from the page. The details of your schema are conveniently presented for reference as well. To accompany this new interface, the term Assemblies has been simplified to Database in the UI. Assemblies remain the underlying concept tying components together, and are still referenced in documentation. Manual upgrade of old streams and schemas may be required. See upgrading entities in the UI for more details.

Queries

To protect against reserved word and column name conflicts, the way labels are passed to a query has changed. Labels should now be specified either as a dictionary, or with the prefix label_. The old label style is still supported in 1.5, but is deprecated and will be disabled by default in 1.6. See upgrade considerations on labels for more details.
Distinct has been added to the available aggregations in getData for database queries.

Pipelines

Users can now use Python code in all areas of the pipeline UI, allowing for further flexibility in Python user experience with kdb Insights Enterprise. Previously, the addition of Python code to sections of pipeline UI were limited in scope to the function cells of the pipeline configuration forms.
The Stream Processor Python API can now leverage automatic schema conversion using a new function kxi.sp.transform.schema.
Users can now use the Kafka Schema Registry in Python and the UI. See kxi.sp.read.from_kafka for Python, and the Kafka reader.
String manipulation utilities have been added to the UI and the Stream Processor Python and q APIs, allowing users to selectively convert the string data to upper or lowercase representations for the purpose of data cleansing.
Non-UI deployments of Stream Processors now support pipeline replicas, providing enhanced resilience to deployed pipelines. Users can now choose to run their pipeline with replicas for high availability in the Stream Processor. This replaces the previous use of pipeline groups.

Reliable Transport

Reliable Transport now offers subscribers the ability to use dynamic client-side filtering, enabling users to control what parts of the data feed their subscribers need to receive. This can greatly improve ingestion performance by eliminating the need for subscribers to handle discarding of unnecessary data. See Filtering for more information.
RT has introduced logHistory, a new API to assist in analysis and debugging. The API returns metadata about the RT log file history for logs that are no longer available. This includes:
- The log file name.
- The time of the first message added to that log file.
- The position of the first message added to that log file.
See Log File Meta Data for more information.

Command Line Interface

A rollback command has been added to the CLI to simplify rolling back your installed instance of kdb Insights Enterprise after upgrading.
A kxi user subcommand has been added to the CLI to support managing users programmatically. This provides greater flexibility than using the keycloak initUser values at deploy time.

Standalone Installation

Ability to control which components should be enabled (cert-manager,ingress-nginx, sharedfiles storage class). The available components are defined on the configuration file (kxi-terraform.env).

Improvements

Batch ingest now supports ingesting tables that have no symbols. Additionally, extra validation has been added to verify symbol integrity.
Batch ingest now supports sorting of columns.
The kdb Insights Database gateway now supports returning responses that are greater than 2GB for IPC requests. See routing configuration for more details.
Database schemas in the UI can now be modified for assemblies that have been stopped. See schema modification.
Requests that supply a log correlator (logCorr) will now see that correlator in the Service Gateway logs. Previously a random correlator was used in the Service Gateway logs.
Users can now utilise backtick syntax when defining aggregations in Views, e.g. `c1`avg`price.
Pipeline editors can now be resized vertically.
Labels for improved accessibility have been added to the Pipelines screen.
Added significant improvements for accessibility using keyboard navigation and/or screen readers in the UI.
- Users can exit a text area in a code block on the Query page by first pressing "CTRL+M" before pressing the "Tab" key.
- The entity tree menu button can now be opened when hovering your mouse over the entity tree. Arrows can then be used to navigate.
- "Code View" on the Schemas page can now be closed with the "Esc" key.
- The Views and Pipeline pages now allow draggable elements to be inserted/dragged onto the page with the use of your keyboard.
Kubernetes 1.25 is now supported and the default version for standalone installations.
Ensure the latest AWS CLI is installed on the docker image used for standalone installations

Fixes

Expand to see the full detail of resolved issues here

Database

Additional historical tiers that used object storage in the UI now allow dashes and spaces in file names. Previously they would error for dashes and spaces.
Migrating tiers to object storage could fail with the eventual error SMDBM Migration discontinued due to failure if multiple threads were used.
New validation has been added to prevent accidentally adding a object storage tier to a database that does not already have one. This would previously error at EOD with a file not found error.
Schema conversions are now more robust on network based file systems (ex. Rook Ceph). Previously, this could result in a EXDEV. OS Reports: Invalid cross-device link error in the Storage Manager.
Changes to disk sort columns in a schema that would be incompatible with the current disk sort will now error with a clear message at deploy time. Previously, this would error with a u-fail error either at an EOI or EOD.
Batch ingest can now gracefully handle scenarios where there is no data provided in the session. Previously, this would result in the following error: Error reporting to licensing server: Value must be a long, leaving the ingest pipeline stuck in a RUNNING state.
When removing a symbol column from a table, the SM should be able to properly handle its removal as well as the sym file.
If an in-memory sort is applied to a column in a schema, it is now sorted on arrival in the RDB. Previously, if data arrived out of order with this set, an s-fail error would be thrown.
In the Storage Manager metrics, the kxi_sm_hdb_date_records metric no longer reports an empty value at startup, which resulted in false alerts about the HDB being offline.
The Storage Manager now double buffers its state table in order to protect against corruption.
Rapid successive queries using qsql, such as those from Views in the kdb Insights Enterprise UI, no longer have the potential to return query results out of order.
The kxi_sg_pending metric would previously be decremented twice when a connection closed, resulting in negative pending values.
The Service Gateways were creating duplicate connections to Resource Coordinators while they were waiting to come online. This was resulting in a large number of redundant connections being opened between the services. In large deployments, this could result in conn errors in the Resource Coordinator.
RDBs and HDBs that require a dependency on the IDB will now error at startup if one is not specified as part of the mount configuration.

User Interface

Previously, a database would show a loading spinner if one of its pipelines had reached a ‘finished’ state while its other components were in a ‘ready' state. This has now been rectified and the database will display a ‘ready’ state in this case.
An issue with pasting into text fields in a pipeline node’s configuration has been fixed. Previously, the node may be been duplicated.

Stream Processor

Pipeline names are now properly displayed in the Running Pipelines section of the Overview page. Previously, these were displaying the pipeline ID and were not clickable for viewing further information.
Pipelines deployed via as part of a Database now properly set Kubernetes secrets. Previously, secrets were only being set properly for standalone pipelines.
The presence of a UTF-8 byte-order mark at the beginning of CSV files will now be automatically removed from data when acted on by the Stream Processor CSV decoder.
Expression readers now return the result of Python lambdas.
Window nodes in the Stream Processor can now accept dictionaries as input batches. Previously, incoming dictionaries would be merged with input batch in the buffer, instead of appending to a buffered list of dictionaries.
The python write.to_database node previously converted assembly names to booleans. This is now resolved, and also fully supports the direct write functionality.
The Stream Processor's Protobuf decoder no longer ignores the value of As List when a pipeline is deployed from the UI. Previously, this value was being overwritten.
The Stream Processors Protocol Buffer Encoder node now properly saves the selected Payload Type.
The Stream Processor Database Reader now supports SQL statement queries. Previously, this node had erroneously suggested the use of qsql statements for querying which was unsupported.
The default onFinish functions for the apply node have been updated to have the correct parameters and examples of flushing a buffer.
When using directWrite functionality in .qsp.write.toDatabase, table names as strings are now supported. Previously, this had to be a symbol.
Copy/pasting nodes within a pipeline document while other pipeline documents are open in the background no longer produces duplicates. Previously, the copy/paste operation was being applied to both the intended pipeline as well as the unintended pipelines, resulting in nodes disappearing due to shared names/ids.
Security vulnerabilities patched in the Stream Processor for the following CVEs: CVE-2022-41910, CVE-2022-41902, CVE-2022-41900, CVE-2022-41880.
Query sessions had previously set error trapping to 1 within the q session, resulting in users being unable to interact with their query session. This behaviour manifested in users being unable to interact with their query sessions until they were manually restarted.

Command Line Interface

With previous versions of the kdb Insights CLI, the kxi install list-versions command would return an outdated list of versions. This has now been resolved and the list will be updated on each invocation.

Standalone Installation

Fix Rook-Ceph OSD not getting assigned if underlying node changes

Third-party dependencies

kdb Insights Enterprise 1.5.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.5.0 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.5.0.tgz
Operator	kxi-operator-1.5.0.tgz
CLI	kxicli-1.5.0-py3-none-any.whl
C SDK	kxi-c-sdk 1.5.0
Java SDK	kxi-java-sdk 1.5.0
RT Bridge	rt-bridge 1.5.0
Infrastructure	kxi-terraform-1.5.0.tgz

Known Issues

LicensingCLIQueriesPipelinesRollbackReinstall

On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.
A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below
```
klic license renew $LICENSE_UUID --version=latest

env QLIC=$(pwd) klic license fetch $LICENSE_UUID
```
Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.
```
printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml

kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml
```

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. Therefore, there is a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:
- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit super back to sp
- Press the down arrow to go to the next line

Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

A known issue exists within the kxi install rollback command when rolling back to prior versions. When the target version was installed with keycloak.initUser enabled. It is expected the keycloak-config-cli job will fail. This is due to the rollback trying to revert that user's password and violating the password history policy.

The rollback failure will manifest as the failure of the keycloak-config-cli job. This will create job pods as it fails and retries. In order to proceed with the rollback it's recommended to delete kubectl delete job ${RELEASE_NAME}-keycloak-config-cli.

A known issue exists during a reinstall using the CLI where the keycloak.initUser is enabled. If the user's password has changed, the reinstall will try to revert it back and be rejected by the password history policy.

This can be avoided by disabling the user import on reinstall using --import-users False as below:

kxi install run --import-users False ..

Third-party dependencies

kdb Insights Enterprise 1.5.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.5.0 release of the standalone infrastructure installation scripts.

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly from 1.3.x to any newer release, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

Stop all external publishers to kdb Insights Enterprise
Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

Backup any assemblies that may be running

$ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

Teardown assemblies named in the output of the backup command

$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

Delete the RT PVCs

kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

Carry out the upgrade

Restore assemblies

$ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

Restriction on upgrading to older versions

The CLI will not allow you to "upgrade" to an older version of kdb Insights Enterprise. Please use the rollback command to revert to a previous version.

Backward Compatibility

If you are upgrading from a version earlier than 1.5.0 you must also read each of the relevant release notes to ensure you know all the changes that will affect the upgrade. Refer to the previous release page for older releases.