kdb Insights Enterprise - Previous Release Notes

This page outlines release notes from past releases. For the most recent release, refer to the latest release page.

1.10.1

Release Date 2024-07-24

Fixes

Expand to see the full detail of resolved issues here

Packaging

Resolved an issue in version 1.10.0 where, when removing a pipeline from a package using the UI, the pipeline would revert to a default pipeline specification instead of being removed.
Resolved an issue where adding a pipeline into two databases caused the UI to not behave correctly.

Database

Fixed an issue where getData queries would fail with a type error if the time range of the request landed completely between two IDB partitions.
An embedded assembly Resource Coordinator could become unreachable if its connection with the Service Gateway dropped.
Batch ingest has been modified to reduce the amount of RAM required for re-enumeration and sorting during a batch ingest operation.
Fixed an issue where batch ingest in directWrite mode did not properly apply primaryKeys to the data. Prior to this release, the data would not be properly collapsed, resulting in multiple rows for each primary key being persisted to disk. See the following example:

The data was ingested using batch ingest with `directWrite` mode enabled into table `t` that has primary keys configured as `pk1`, `pk2`

pk1 pk2 val
-----------
a   1   0
a   1   1
b   1   2
b   2   3
b   1   4
c   4   5

Prior to this release, the below data would be persisted to disk in the HDB:

pk1 pk2 val
-----------
a   1   0
a   1   1
b   1   2
b   2   3
b   1   4
c   4   5

The behaviour has been fixed to correctly apply the primary keys to the data, meaning that the below rows would now be persisted to the HDB:

pk1 pk2 val
-----------
a   1   1
b   1   4
b   2   3
c   4   5

If tables are configured with primary keys, then after upgrading to 1.10.1, you may notice a decrease in the number of records being persisted when compared to data in older partitions. This is due to the corrected behavior.

If you have data from a prior release that was ingested via batch ingest with directWrite mode enabled, it is possible to correct it. For assistance with this, contact KX Support.

Readers

Fixed an issue where the Stream Processor database reader would throw an error when any filters were specified.

Pipelines

In version 1.10.0, pipelines with an Apply node(s) from releases older than version 1.9 would fail on deployment. This has been resolved, and older pipelines that use Apply nodes now work as expected without any user action required.

Third-party Dependencies

kdb Insights Enterprise 1.10.1 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.10.0 release of the standalone infrastructure installation scripts.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type	Nexus location	Downloads Portal location
Enterprise	insights-1.10.1.tgz	insights-1.10.1.tgz
Operator	kxi-operator-1.10.1.tgz	kxi-operator-1.10.1.tgz
CLI	kxicli-1.10.1-py3-none-any.whl	kxicli-1.10.1-py3-none-any.whl
RT C interface	kxi-c-sdk 1.10.0	kxi-c-sdk 1.10.0
RT Python interface	kxi-rtpy-1.10.0	kxi-rtpy-1.10.0
RT Java interface	kxi-java-sdk 1.10.0	kxi-java-sdk 1.10.0
RT Bridge	rt-bridge 1.7.0	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.10.1.tgz	kxi-terraform-1.10.1.tgz
kxi-management-service	kxi-management-service-0.2.4.tgz	kxi-management-service-0.2.4.tgz
insights-on-k8s	insights-on-k8s-0.1.6.tgz	insights-on-k8s-0.1.6.tgz

1.10.0

Release Date 2024-06-20

The main focus of this kdb Insights 1.10 release is a set of incremental updates designed to simplify system administration and streamline user experience.

New Features

SQL2 now supports joins
Standardized Auditing
Environment Variables in Packages
kxi-python now supports publish, query and execution of custom APIs
Publishing to Reliable Transport (RT) using the CLI
Stream Processor supports subscribing to the end of an RT stream
Schema Conversion Progress Indicator
Natural language descriptions included in getMeta descriptions
Cost Reduction Options for Reliable Transport

1. SQL2 now supports joins

Joins support in SQL2 enables you to combine multiple tables and/or dictionaries in your workflows using either INNER, LEFT, RIGHT, FULL, or CROSS joins. To find out more about streamlining your data connectivity across datasets, refer to the joins documentation.

2. Standardized Auditing

Standardized auditing provides reliable and consistent event logging, bolstering system security, and event accountability. The new structured framework introduced in this release adapts to current and future auditing needs aligned with specific services, to ensure comprehensive monitoring and robust accountability. Find out more about the audit framework in the Logging documentation.

3. Environment Variables in Packages

Environment Variables in Packages enables administrators to inject environment variables at runtime and dynamically pass values to both the database and pipelines. Variables can be set globally or per component and are applicable for custom analytics through global settings. Learn how to implement environment variables in the Package deployment documentation.

4. kxi-python now supports publish, query and execution of custom APIs

The Python interface to kdb Insights Enterprise, kxi-python, has been extended to allow for publishing to your kdb Insights Enterprise deployment. Refer to the documentation as a guide for how to both publish and query data using it. kxi-python also now supports the execution of custom APIs against your deployment, as shown in this example. These enhancements promote a stronger developer experience by significantly improving efficiency and streamlining workflows.

5. Publishing to Reliable Transport (RT) using the CLI

You can now use KXI CLI to publish ad-hoc messages to kdb Insights Enterprise using Reliable Transport. This change empowers you to use tools such as VS Code to publish data directly to the database, simplifying integration and real-time insights. This documentation demonstrates how to publish data using the CLI to kdb Insights Enterprise.

6. Stream Processor supports subscribing to the end of an RT stream

This release introduces the ability to subscribe to the end of an RT Stream. When configuring the kdb Insights Stream reader, it now provides an end parameter which will subscribe to the latest position in the stream instead of the beginning. This enables late joiners to reduce consumption and enhancing operational efficiency. Refer to the documentation links to learn more:

7. Schema Conversion Progress Indicator

A new schema conversion progress indicator provides data engineers and developers with visibility of the schema conversion process, including progress and error reporting.

This is a useful tool for monitoring the progress of schema conversions, especially for large data sets which require a significant amount of time.

8. Natural language descriptions included in `getMeta` descriptions

getMeta descriptions now include natural language descriptions of tables and columns in its result, enabling you to attach and retrieve detailed descriptions of database structures. Read more in our documentation.

9. Cost Reduction Options for Reliable Transport

An alternative topology for a publisher to choose when sending data to RT has been added. This alternative has the advantage of reducing the ingress bandwidth to a third of what the default topology requires. This could offer significant cost savings where ingress data incurs charges. Refer to the documentation for details.

Improvements

Performance Improvements in getData

For queries that span multiple partitions, getData has been optimized to improve the query execution times.

End-of-Interval Memory Optimization

End-of-Interval (EOI) Memory Optimization ensures that large, splayed tables are automatically cleared at EOI. This enhances memory efficiency and reduces RAM dependency, resulting in accelerated system performance and smoother system operations. There is no user configuration required to take advantage of this enhancement. See the documentation for information on splayed tables.

Updated Entitlement Administration

The commands to use entitlements with the CLI have changed. Refer to Configuring data entitlements for details.

getData REST Request Filtering

REST requests in getData now support filtering on time, minute, and timespan data type fields.

Streaming JSON responses over HTTP in the Service Gateway

The Service Gateway (SG) now supports returning JSON responses in a streaming fashion without the need to compress the result. This helps reduce the necessary amount of RAM the SG needs in order to handle larger payloads. The environment variable KXI_SG_STREAM_THRESHOLD will allow you to configure your chosen chunk size for splitting up the overall response. This capability is automatic in the Service Gateway and does not require any user configuration. See the documentation here for more details.

Customizable Polling Frequency in File Watchers

Cloud-polling file watchers now support the ability to configure the frequency in which polling for new files is done. When configuring a pipeline, the Use Watching checkbox provides a Timer option for selecting a frequency to poll by. See the documentation describing the optional parameter.

Stream Processor Kafka Writer

The SP Kafka Writer now supports an options parameter for specifying advanced configuration on the Kafka broker connection. Previously, it only supported advanced configuration on the topic via the topicConfig parameter. See the documentation for information.

Filtering Rows in Views

Data sources in Views now support the Max Rows option that limits the maximum number values displayed.

Select Columns

In the UI Query screen, it is now possible to filter a query by a selected column(s) to refine the data returned by the query. This allows users to reduce payload sizes and avoid potentially running out of RAM.

Streaming to Views

Streaming data sources now include a Max Rows option which facilitates the use of a rolling window chart that shows the last x number of points.

The Streaming to Views walkthrough has been updated with a new section Filtering based on a dropdown to show how to filter a Data Grid to a specific value from a list.

Fixes

Expand to see the full detail of resolved issues here

Database

For a new system with an empty database, queries would not work after the first end-of-day data writedown if no data had been ingested yet. This issue has been resolved.
Previously, a user would need to manually edit their par.txt file if the object storage location information was changed. This has been resolved so that par.txt is updated automatically after completing the required restart.
The UI was erroneously reporting that a schema change was in progress in cases where an error in conversion had occurred. This issue has been resolved.
When using nested filters in getData on keyed partitioned tables with late data, updates to on-disk data were not being reflected in the output of getData, if the data had not yet been written to disk. This issue has been resolved.
Importing a database from object storage is now correctly sourcing the database's sym file upon initialization of the Storage Manager.
Previously, getData REST queries would produce inf instead of a null. This inf response would break most JSON parsers. This issue has been resolved.
Prior to 1.10.0, the Service Gateway was configuring the minimum heap size to retain equal to the container requests. This could result in the container holding onto unused memory and not releasing it back to the OS. Additionally, the gateway was not setting an explicit size for off-heap memory used by IO messages, which could result in using a combined total of heap and off heap memory exceeding the container limit.

To avoid both of these issues, set JAVA_OPTS explicitly. Use -Xms to set the minimum heap size to always retain, -Xmx to set the maximum heap size, and -XX:MaxDirectMemorySize to set the maximum size used for IO messages off heap. We recommend splitting the container limits equally between heap and IO.

Example configuration of 256Mb minimum reserved heap and 2G limit, split into 1g for heap and direct memory:
```
`JAVA_OPTS="-Xms256M -Xms1G -XX:MaxDirectMemorySize=1G"`
```

Reliable Transport

Previously, the RT client could fail to recognize a new log file, forcing downstream subscribers into a hung state. This has been resolved.

UI

On the Resource tab, incorrect database resource summary values were displayed when no limits were specified. Now, if the limit value is empty, the request value is used to calculate summary values.
Previously, the /details API didn't always show pipelines that were still in a TEARING_DOWN status.
When configuring a pipeline in the UI, setting additional workerThreads did not correctly apply the specified configuration. However, setting the environment variable KXI_DEFAULT_SP_WORKER_THREADS did work. This has been resolved so either method can be used.

Cloud Readers

A file watcher in Azure reader did not work for test or full deploys.
An issue has been resolved where, when administering a pipeline, a cloud reader node (s3, ms, and gs) could prompt you that there were unsaved changes even when no changes have been made.

Packages

Package names created in the CLI are now restricted to only use lower case letters and limited to 24 characters, inline with the GUI and Kubernetes restrictions. Previously, the deployment would fail because Kubernetes was unable to deploy an assembly with capital letters in its name.
Previously, exporting a pipeline did not complete successfully. This is now fixed.

Kubernetes Discovery

Previously, the KXI_RC_LABEL_SELECTOR field was not being correctly honored, causing Kubernetes discovery to become non-functional.

Scratchpads

Using a file watcher (ex. await pending file) was not working in a Scratchpad session.

Security

Various CVEs have been remediated as part of this release.

Third-party Dependencies

kdb Insights Enterprise 1.10.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.10.0 release of the standalone infrastructure installation scripts.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type	Nexus location	Downloads Portal location
Enterprise	insights-1.10.0.tgz	insights-1.10.0.tgz
Operator	kxi-operator-1.10.0.tgz	kxi-operator-1.10.0.tgz
CLI	kxicli-1.10.0-py3-none-any.whl	kxicli-1.10.0-py3-none-any.whl
RT C interface	kxi-c-sdk 1.10.0	kxi-c-sdk 1.10.0
RT Python interface	kxi-rtpy-1.10.0	kxi-rtpy-1.10.0
RT Java interface	kxi-java-sdk 1.10.0	kxi-java-sdk 1.10.0
RT Bridge	rt-bridge 1.7.0	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.10.0.tgz	kxi-terraform-1.10.0.tgz
kxi-management-service	kxi-management-service-0.2.3.tgz	kxi-management-service-0.2.3.tgz
insights-on-k8s	insights-on-k8s-0.1.6.tgz	insights-on-k8s-0.1.6.tgz

Upgrade/Rollback Notes

New Requirements for Hostnames & kxi package

When you run a kxi package command, the kxi cli first authenticates against kdb Insights Enterprise. In 1.9, running a kxi package command would look up the cached hostname that had been previously authenticated against in the ~/.insights/credentials file. In 1.10.0, the hostname previously authenticated against is no longer used. Instead, you must set the hostname in your ~/.insights/cli-config.

Views Impacted On Rollback

If you perform a rollback from version 1.10 to 1.9.x, all Views disappear from the UI due to the package manifest file being changed from 'reports' to 'views' in 1.10. The workaround is to pull the package from the package-manager server, manually change 'views' to 'reports' and push the package back to the server.

kxi package pull <package>/<version>
kxi package unpack <package>-<version>.kxi
mv <package>/views <package>/reports
sed -i 's/views:/reports:/g' <package>/manifest.yaml # Can do this step manually in editor
sed -i 's/views\//reports\//g' <package>/manifest.yaml # Can do this step manually in editor
kxi package packit <package> --tag
kxi package push <package>-<version>.kxi

Shutting Down Databases Before Upgrading

The operational procedure described below is no longer required for an upgrade to 1.10.0. However, it is still true for upgrading from 1.8 to 1.9.

Before you upgrade to 1.9.x, if you have any running databases, it is advised to manually stop them before upgrading, otherwise you may become unable to edit them in the UI. If this happens, you can recover the editable state by considering the following options:
- If possible, back up the running assembly yaml file with kubectl get asm
- Tear down the running assembly manually with kubectl delete asm
- Optionally, use the kxi package deploy command in the CLI, specifying the name and version of the package. The package appears under kxi package remote-list. Refer to Package deployment documentation for more information.
If the package doesn't reappear as editable after tearing down, you need to use the KXI CLI to perform the following operations:
- Convert the assembly yaml to a package. Refer to Converting assemblies to packages.
- Upload the package to kdb Insights Enterprise. Refer to Uploading a package.
- Deploy the package from the UI. Refer to Package deployment.

Known Issues

Expand to see the list of known issues here

Resiliency With Resource Coordinators Deployed via kxi package

If the Resource Coordinator (RC) is deployed using kxi package, and a connection is lost between the Service Gateway (SG) and the RC, queries routed to said RC will stop being serviced. This does not occur in cases where the deployment was done with the base Insights Enterprise install.

If the connection is lost and queries are failing, this can be resolved by restarting the Service Gateway.

PostGreSQL & Keycloak Resiliency

A transient issue can occur in which Keycloak is unable to re-establish a connection to PostgreSQL after it recovers from an outage. This can result in getData being unable to serve queries. To work around this, restart PostgreSQL.

Pipelines

Pipelines that use Apply nodes from releases older than 1.9 will fail on deployment after upgrading. To work around this, after performing your upgrade, open the pipeline, click on Apply node, click the gear icon in the top corner of the node-config area, and then set the parameters manually (for most apply nodes this will be operator, metadata, data).
Deploying a pipeline from the UI will mark it as having unsaved changes even when none exist. Clicking Save again will make this go away.

Upgrades & Rollbacks

Users are prompted to tear down assemblies upon rollback. The rollback should exit if the user responds ‘n' to this prompt. Instead, the rollback continues.
During upgrade, the postgresql process may log errors similar to the below. These can be safely ignored.
```
`FATAL: database `[database name]` does not exist`
```

UI

Changing the system log level in the UI does not get actioned.

Packaging

In the UI, when you remove a pipeline from a database, packaging preserves this pipeline in a separate package to ensure it is not permanently deleted. This means the pipeline is still available in the UI. However, if you re-push the package from the CLI that still contains the pipeline, the pipeline now exists in two packages, and the latest version of the pipeline is not being served to the UI. The workaround is to remove the pipeline-only package that has been created.

Beta Features

Streaming to Views has transitioned out of Beta and into general availability.

Summary

We hope you find some useful features that optimize your kdb Insights Enterprise experience. Try them out and email our Support Team if you need any help.

We look forward to bring you even bigger features in kdb Insights Enterprise 1.11 coming soon!

1.9.4

Release Date

2024-06-14

Fixes

Expand to see the full detail of resolved issues here

Security

Multiple CVEs have been resolved as part of this patch.

Cloud Readers

When administering a pipeline, a cloud reader node (s3, ms, and gs) could prompt you that there were unsaved changes even when no changes have been made.
The Azure reader file watcher was returning an error that a file(s) did not exist despite the file(s) being present.

Reliable Transport

RT's replicator was emitting an Invalid publishing directory, was it deleted by mistake? Please enrol a different client and restart the SDK with the new client error after recovering from a power outage of a q publisher server. This has been resolved to handle the recovery cleanly after such an outage.

UI

On the Pipeline page, an intermittent issue could cause performance degradation or unresponsiveness in the UI for extended periods.
When configuring a pipeline in the UI, setting additional workerThreads was not correctly applying the specified configuration. However, setting the environment variable KXI_DEFAULT_SP_WORKER_THREADS did work. This has been resolved so either method can be used.
The default setting for retries for database reader/writers in the UI did not match the backend defaults. This has been corrected to be consistent, with the default setting of 60 seconds for retries, and a retry wait time of 3 seconds.
When performing a rollback, the CLI was deploying the rollback task with a hard-coded timeout. This has been changed to a configurable value for use when rollbacks are expected to take longer.
When deploying a database, the Deploy popup was displaying totals that did not match the selected CPU, RAM and disk settings. The same issue was present in the Resource Summary box on the main page. These calculation errors have been resolved to display correct totals.

Database
For a new system with an empty database, queries would not work after the first end-of-day data writedown if no data had been ingested yet.

Third-party Dependencies

kdb Insights Enterprise 1.9.4 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.9.1 release of the standalone infrastructure installation scripts.

Artifacts //! devops to confirm versioning

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type	Nexus location	Downloads Portal location
Enterprise	insights-1.9.4.tgz	insights-1.9.2.tgz
Operator	kxi-operator-1.9.1.tgz	kxi-operator-1.9.1.tgz
CLI	kxicli-1.9.0-py3-none-any.whl	kxicli-1.9.0-py3-none-any.whl
RT C interface	kxi-c-sdk 1.9.0	kxi-c-sdk 1.9.0
RT Python interface	kxi-rtpy-1.9.0	kxi-rtpy-1.9.0
RT Java interface	kxi-java-sdk 1.9.0	kxi-java-sdk 1.9.0
RT Bridge	rt-bridge 1.7.0	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.9.1.tgz	kxi-terraform-1.9.1.tgz

1.9.3

Release Date

2024-05-21

Fixes

Expand to see the full detail of resolved issues here

Reliable Transport

RT's replicator was emitting an Invalid publishing directory, was it deleted by mistake? Please enrol a different client and restart the SDK with the new client error after recovering from a power outage of a q publisher server. An intermittent issue with truncated log files when the publisher restarts has been resolved.

Third-party Dependencies

kdb Insights Enterprise 1.9.3 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.9.1 release of the standalone infrastructure installation scripts.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type	Nexus location	Downloads Portal location
Enterprise	insights-1.9.3.tgz	insights-1.9.3.tgz
Operator	kxi-operator-1.9.1.tgz	kxi-operator-1.9.1.tgz
CLI	kxicli-1.9.1-py3-none-any.whl	kxicli-1.9.1-py3-none-any.whl
RT C interface	kxi-c-sdk 1.9.0	kxi-c-sdk 1.9.0
RT Python interface	kxi-rtpy-1.9.0	kxi-rtpy-1.9.0
RT Java interface	kxi-java-sdk 1.9.0	kxi-java-sdk 1.9.0
RT Bridge	rt-bridge 1.7.0	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.9.1.tgz	kxi-terraform-1.9.1.tgz
kxi-management-service	kxi-management-service-0.2.1.tgz	kxi-management-service-0.2.1.tgz
insights-on-k8s	insights-on-k8s-0.0.16.tgz	insights-on-k8s-0.0.16.tgz

1.9.2

Release Date

2024-05-07

Fixes

Expand to see the full detail of resolved issues here

Reliable Transport

The RT client was failing to recognize a new log file, putting downstream subscribers into a hung state. This has been resolved.

Third-party Dependencies

kdb Insights Enterprise 1.9.2 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.9.1 release of the standalone infrastructure installation scripts.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type	Nexus location	Downloads Portal location
Enterprise	insights-1.9.2.tgz	insights-1.9.2.tgz
Operator	kxi-operator-1.9.1.tgz	kxi-operator-1.9.1.tgz
CLI	kxicli-1.9.0-py3-none-any.whl	kxicli-1.9.0-py3-none-any.whl
RT C interface	kxi-c-sdk 1.9.0	kxi-c-sdk 1.9.0
RT Python interface	kxi-rtpy-1.9.0	kxi-rtpy-1.9.0
RT Java interface	kxi-java-sdk 1.9.0	kxi-java-sdk 1.9.0
RT Bridge	rt-bridge 1.7.0	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.9.1.tgz	kxi-terraform-1.9.1.tgz

1.9.1

Release Date

2024-04-30

Improvements

The fromParquet Reader is now supported for AWS-based deployments. For more information, refer to Readers.
Streams can now be removed from packages using the rm command from the CLI. Run kxi package rm --help in the CLI for details.

Fixes

Expand to see the full detail of resolved issues here

Database

Client requests using compressed IPC messages are now properly serviced. Previously, if the first request from a client was using compressed IPC, that and any future requests would fail.

Terraform Scripts

Previously, in the kdb Insights Enterprise Terraform sample client scripts, the package-manager-migration command failed when used in Azure deployments using the sharefiles storage class. This has been resolved. New sample scripts are available as part of this release. See the Artifacts section below.

Stream Processor

Pipelines now throw an error when writing to a non-existent table. Previously, no errors were present in the pipeline or database logging. This was most commonly experienced when renaming a table in the database without updating the pipeline as well.

Reliable Transport

Reliable Transport pods now use Kubernetes liveness probes to automatically bounce pods if unresponsive. This resolves a known issue in 1.9.0 in which RT pods were required to be manually restarted if an upgrade erroneously left them in a non-ready state.

Packaging

The UI now displays the running version of a package as opposed to the latest package that may be available.
subTopic has been added as a mandatory field when adding external streams to a package. Previously, this was not present and caused a package to be undeployable.
In the UI, it is no longer possible to create pipelines with the duplicate names.
The CLI now errors when trying to create and save a package that contains duplicate UUIDs, or when trying to save entities within a package that have duplicate UUIDs.

UI

Resource allocations are no longer mandatory input fields on the Resources tab. It is now possible to leave them empty and submit your resource configuration. For more information, refer to Adjust resources.

Third-party Dependencies

kdb Insights Enterprise 1.9.1 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.9.1 release of the standalone infrastructure installation scripts.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type	Nexus location	Downloads Portal location
Enterprise	insights-1.9.1.tgz	insights-1.9.1.tgz
Operator	kxi-operator-1.9.1.tgz	kxi-operator-1.9.1.tgz
CLI	kxicli-1.9.0-py3-none-any.whl	kxicli-1.9.0-py3-none-any.whl
RT C interface	kxi-c-sdk 1.9.0	kxi-c-sdk 1.9.0
RT Python interface	kxi-rtpy-1.9.0	kxi-rtpy-1.9.0
RT Java interface	kxi-java-sdk 1.9.0	kxi-java-sdk 1.9.0
RT Bridge	rt-bridge 1.7.0	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.9.1.tgz	kxi-terraform-1.9.1.tgz

Upgrade Notes

Upgrading to 1.9.x

Before you upgrade to 1.9.x, if you have any running databases, it is advised to manually stop them before upgrading, otherwise you may become unable to edit them in the UI. If this happens, you can recover the editable state by considering the following options:
- If possible, back up the running assembly yaml file with kubectl get asm
- Tear down the running assembly manually with kubectl delete asm
- Optional: use the kxi package deploy command in the CLI, specifying the name and version of the package. The package should appear under kxi package remote-list. Refer to Package deployment for more information.
If the package doesn't reappear as editable after tearing down, you need to use the KXI CLI to perform the following operations:
- Convert the assembly yaml to a package. Refer to Converting assemblies to packages.
- Upload the package to kdb Insights Enterprise. Refer to Uploading a package.
- Deploy the package from the UI. Refer to Package deployment.

Known Issues

Expand to see the list of known issues here

Custom Tier Names

Existing assemblies that have non-default tier names (i.e. names other than "rdb","idb","hdb") that are converted to packages will need to be updated to use those defaults. Presently, the UI expects the default names and is unable to differentiate between hdb and additional tiers.

1.9.0

Release Date

2024-03-26

New Features

Developer Experience Improvements for Packaging

kdb Insights Enterprise packaging now allows clients to manage database and pipeline configuration as well as custom code within a single package. This greatly simplifies developer and operational workflows, as users can maintain and administer databases and pipelines within a single package and with specific configuration for each pipeline. Packaging has also been integrated with the UI so that any databases and pipeline created using the CLI using the kxi package command can be viewed and maintained by the UI and vice versa. Refer to Packaging for more details.
SQL Enhancements

SQL in kdb Insights Enterprise now includes support for ORDER BY and LIMIT clauses, in addition to other functions. Changes to how SQL statements are parsed also significantly improves SQL query performance in this release. Query performance, on average, is up to 3x faster than before. Refer to SQL 2 for more information on the new SQL implementation and its current Known Limitations.
Streaming to Views

In the kdb Insights Enterprise UI, Views can now subscribe to real-time data from a pipeline, rather than relying on intermittent polling. This gives you the ability to visualize and analyze your data immediately. Refer to the streaming guided tour for details on configuring pipelines and views for this feature.

Streaming to KX Views is a BETA feature in kdb Insights Enterprise 1.9.0.

· Refer here to the Standard Terms related to Beta features.

· We invite you to use this Beta feature and to provide feedback using the portal.

· Refer here to the known limitations of this Beta feature.
Snapshots & Majority Archival for Reliable Transport

Reliable Transport now supports taking snapshots allowing for majority archival of logs files. Previously, all nodes needed to be healthy before RT logs could be archived.
Configurable Connections for Reliable Transport

Reliable Transport now supports subscribers and publishers setting customized parameters when they register with their respective sequencer cluster. This gives you greater configurability in how you deploy and orchestrate components of RT. See rt.qpk for more details.

Improvements

System Resources Configuration

Defining system resources has been streamlined, enabling you to set all aspects of resource allocation from a single configuration page in the kdb Insights Enterprise UI. See Adjust Resources for more information on this new tab.
Batch Delete

Batch delete now allows you to delete from all tiers of your HDB. Previously, this was only possible for the tier with the most recent HDB data. See Database Delete for details on how to perform a delete.
Python Cloud Readers

Python cloud readers now support polling for new matches to a glob pattern with the watch parameter. See Readers for details.
Upgrade to Insights Core 4.1.1

kdb Insights Enterprise images have been upgraded to run Insights Core 4.1.1, which runs kdb+ 4.1 2024.03.12.
Upgrade to RockyLinux 9

kdb Insights Enterprise images have been upgraded to run on Rocky Linux version 9 as their base operating system. This addresses security vulnerabilities present in version 8.
Stream Processor Troubleshooting Guide

There is now a troubleshooting guide{:target="blank"} available to help users better understand how to debug their pipelines running in _kdb Insights Enterprise.

Fixes

Expand to see the full detail of resolved issues here

Database

If EOD failed while in the middle of an HDB transaction, the aborted transaction was not properly cleaned up after the failure and could result in data inconsistency after reconnection.
The EOD process was not performing a garbage collection after a batch ingest.
Data was unqueryable after a batch ingest if no additional streaming data was loaded after the initial batch ingest.

Reliable Transport

In deployments with a one-node RT configuration, RT has been updated to roll its log file after a restart. This prevents data flow from being paused. The RT log file will roll from log.x.y to log.x.y+1.

UI

The Subscriber node was limited to only specifying one keyed column. This has been resolved to allow one or more keyed columns, ensuring that streaming supports a record being streamed per unique combination of all specified keyed columns and to allow filtering on any one of the keyed columns.
When configuring a Schema node, the regular expression pattern for table validation has been updated to allow single character column names.
For Function nodes, users can now customize their function parameters to accept the metadata and operator, in addition to data, in the Pipeline UI. See here for details.
The Error Mode in the Pipeline Settings section of the Pipeline editor has been removed as it is no longer required, due to a stack trace now being logged when user code errors.
The logging level within the Runtime Pipeline Settings section of the Pipeline editor was not correctly displaying or setting the log level of the SP worker. This has been resolved.
There was no way to specify \t as the delimiter in a CSV decoder in the UI. This has been corrected in the CSV Decoder node to allow for any delimiter to be used.
After a pipeline has been deployed, when the status is either “Running” or “Finished”, a user is able to delete the pipeline. When the pipeline is deleted, the Overview page was still displaying it under “Running Pipelines”. This has been resolved to not display deleted pipelines.
Disk sizes for IDB and HDB tiers can now be set to any value greater than 0.
Users without the permission to create a pipeline should not be able to use the Import Data button on Database Page. This has been resolved.
Deleting running databases is now allowed.

Views

Notifications in Views were not reporting correctly. This has been resolved.
Under certain circumstances when you had a particular data source selected in the Data Source window and clicked New the existing values remained visible, however they were all ignored. This has been resolved.
When using the getData API with agg set to sym;distinct;sym, the UI was ignoring this request and not sending any aggregation. This has been resolved.
Switching between tabs caused the user-selected date to switch back to the default setting. This caused the View to constantly reload. This has been resolved.

Stream Processor

When using Python, 'sp.decode.csv' did not support reading columns as strings. This has been updated to enhance flexibility when reading table columns. It now accepts an empty string "" in lieu of a schema to read in all table columns as strings.
Sidecar containers no longer log at a TRACE level by default and now adhere to the log level that is set by the user.
The SP controller now adheres to the log level set by the user.
The CSV decoder did not support decoding csvs which had line-returns (i.e. \n’s) embedded in strings. This functionality can now be turned on with a new optional parameter called Strings Contain Newlines in the UI and newlines in the q and Python APIs.
When running a large import from a static source (file, Azure, AWS, GCP), there was no indication of progress in the logs. The SP logs now displays the file offset periodically while processing to indicate on-going progress of the import.
The SP’s status APIs will now return TEARING_DOWN rather than NOTEXIST when a pipeline has PVCs that are still in a terminating state.
Under certain circumstances when you added an additional stream to your database it could become orphaned after you renamed the database. This has been resolved.
Under certain circumstances when creating a pipeline using the Import feature the Parse Strings column was missing from the Apply Schema page, causing the pipeline to fail when deployed. This has been resolved.
The running pipelines section displays a count of warnings and errors that have occurred in the last 10 minutes. Not all of these warnings were being displayed in the Diagnostics page. This has been resolved.

CLI

When trying to enroll a client with a topic, the client was enrolled but the specified topic was not being set. This has been resolved.
When a user tried to authenticate but did not have a local browser installed on their machine, the redirect URL was localhost instead of the value specified. This has been resolved such that users with a local browser installed can still successfully authenticate.
Installation with a kc.lic license using the --license-filepath command-line parameter was generating an incorrect values.yaml file. This has been resolved so the license values correspond to the license name.
Specifying a different hostname as part of a kxi command is ignored after authentication, as you can only be authenticated to one kdb Insights Enterprise host at a time. For example, if you authenticate to HOSTNAME1 by calling kxi --hostname HOSTNAME1 auth login you are free to make calls against this host. If you attempt to specify a different hostname in subsequent calls kxi --hostname HOSTNAME2, the --hostname parameter will be ignored and the command executed against HOSTNAME1. You must re-authenticate against HOSTNAME2 before running commands against it.
The function .kxi.udfs.list.all[] was failing with a length error. This has been resolved.
Global environment variables set within your values.yaml will now be set by all database and assembly deployments.
```
global:
env:
    env: var
```

Query Environments

Query environments with entitlements enabled required the creation of separate entitlement for the -qe assembly in order for queries to work. This has been resolved.

Upgrades

Fixed an issue where it was possible for pods to be unscheduled after an upgrade or rollback due to a volume node affinity conflict. Please note that this is still possible when rolling back from 1.9 to earlier versions. Please check the release notes of the target rollback version to address.

Security

Various CVEs have been remediated as part of this release.

Third-party Dependencies

kdb Insights Enterprise 1.9.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.9.0 release of the standalone infrastructure installation scripts.

Deprecation

The CLI kxi assembly command is being deprecated in 1.9.0 in favor of kxi package and will be removed in a future version. See kdb Insights CLI Reference for more information on the kxi assembly command.
prtnEndCB and reloadCB callback functions are being deprecated in 1.9.0 and may be removed from future releases. They have been replaced by lifecycle.partition.end and lifecycle.reload.post respectively. Their usage can be seen below.

.custom.da.prtnEndCB:{[event;data]
  log.info("lifecycle.partition.end invoked.")
 }
.custom.da.reloadCB:{[event;data]
  log.info("lifecycle.reload.post invoked.")
 }

.ev.add[`lifecycle.partition.end ;`.custom.da.prtnEndCB];
.ev.add[`lifecycle.reload.post ;`.custom.da.reloadCB];

RT maxLogSize is no longer used in configuration and will be ignored from older configurations. RT will now utilize all disk available to it.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type	Nexus location	Downloads Portal location
Enterprise	insights-1.9.0.tgz	insights-1.9.0.tgz
Operator	kxi-operator-1.9.0.tgz	kxi-operator-1.9.0.tgz
CLI	kxicli-1.9.0-py3-none-any.whl	kxicli-1.9.0-py3-none-any.whl
RT C interface	kxi-c-sdk 1.9.0	kxi-c-sdk 1.9.0
RT Python interface	kxi-rtpy-1.9.0	kxi-rtpy-1.9.0
RT Java interface	kxi-java-sdk 1.9.0	kxi-java-sdk 1.9.0
RT Bridge	rt-bridge 1.7.0	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.9.0.tgz	kxi-terraform-1.9.0.tgz

Upgrade Notes

Keycloak and Postgres Values Changes

The values file settings for the Keycloak and Postgres components have changed for 1.9. The panels below describe the modifications you need to make to this file for all standard or shared Keycloak deployments before upgrading to 1.9.

Merging install values

The values below are a partial extract of your deployment values. These should be merged with your existing values file.

For more information, refer to Applying configuration changes.
Standard DeploymentShared Keycloak
A standard deployment deploys Keycloak as part of the application. If you're using this configuration, you should ensure your values file includes the details shown below.

The names kxi-keycloak and kxi-postgresql are the default values, update these as appropriate.
global: keycloak: auth: existingSecret: kxi-keycloak postgresql: auth: existingSecret: kxi-postgresql .. keycloak: auth: existingSecret: kxi-keycloak postgresql: auth: existingSecret: kxi-postgresql ..
Keycloak values

Please note there are two sets of keycloak and postgres objects; global.keycloak, global.postgresql, keycloak, keycloak.postgresql.

Both sets of these are required as shown above.
If deploying using the shared Keycloak mode, configure the values file as described below.

The KEYCLOAK_RELEASE_NAME and KEYCLOAK_NAMESPACE variables should be updated to match those of your Keycloak deployment. The shared Keycloak docs explain each of these values.
global: keycloak: authURL: http://${KEYCLOAK_RELEASE_NAME}.${KEYCLOAK_NAMESPACE}.svc.cluster.local/auth/ auth: existingSecret: kxi-keycloak postgresql: auth: existingSecret: kxi-postgresql ..
If a user has an existing kxi-license secret created from a kc.lic or k4.lic, kxi install setup will re-use this secret, but will not set the appropriate values in the resulting values.yaml file. See available license types Setup - kdb products.
The default image repository used for new 1.9 installations will be the KX Downloads Portal. The credentials to access this are stored in a docker config json secret named kxi-image-pull-secret. If you are upgrading from an earlier version that used the KX Nexus image repository with an exiting secret kxi-nexus-pull-secret, this existing repository and secret will be used.
After executing an upgrade, if an RT pod does not appear as ‘Ready’, the pod should be deleted/killed so that it is restarted.
During an upgrade, the RT pods attempt to delete a Kubernetes configmap. Since there were 3 RT pods, only one of the pods will be successful in deleting the config map. This means that the other pods will receive ERRORs back indicating that their delete attempt failed. These Failed to delete snapshot: tmp errors are benign and will be set to WARN messages in future.
Query Labels Style

KXI_ALLOW_OLD_LABEL_STYLE now defaults to "false". It must be explicitly set to "true" in the RC to allow the "old label style". For example, in a system with labels region, and class:
- Labels in queries (such as .kxi.getData) must be defined in the labels parameter.
  - old style: --data '{"table": "trade", "region": "usa", "class": "equities"}'
  - new style (new default): --data {"table": "trade", "labels": {"region": "usa", "class": "equities"} }'
- Labels in SQL queries must be prefixed with label_.
  - old style: SELECT * FROM trade WHERE region='usa', class='equities'
  - new style (new default): SELECT * FROM trade WHERE label_region='usa', label_class='equities'

Rollback

If performing a rollback from 1.9.0 to any earlier version, assemblies must be running prior to initiating the rollback to ensure no data is lost. See Upgrading for how to perform upgrades/rollbacks and the available commands to use. Please see the following specific instructions per version:

Rollback from 1.9 to a version less than 1.7 is not possible.

For rollback from 1.9 to a prior version:
1. Confirm all assemblies are running before initiating rollback.
2. Execute the rollback operation. Note that the kxi-rt-rollback and kxi-sp-rollback charts are run automatically as part of the rollback operation as of 1.9.0.
For rollback from 1.8, the following instructions should be followed:
1. Teardown your assemblies.
2. Run the kxi-sp-rollback chart. If the target version for rollback is less than 1.7, also run the kxi-db-rollback chart.
3. Rollback kdb Insights Enterprise per the rollback instructions referenced above.
4. Redeploy assemblies.

Known Issues

Expand to see the list of known issues here

Upgrades & Rollbacks

Users are prompted to tear down assemblies upon rollback. The rollback should exit if the user responds ‘n' to this prompt. Instead, the rollback continues.
Occasionally, rollbacks may fail with an error when running kxi install rollback. This is due to timeouts with the Helm rollback, resulting in errors such as the following:
```
    timed out waiting for the condition
    context deadline exceeded
```
To resolve this issue, re-run the rollback.

Pipelines

It is currently possible to rename a pipeline in the UI to a pipeline name that is already in use. Until this is resolved, ensure your pipeline names are unique.
When deploying a pipeline that writes to a table that does not exist, no error will be thrown.

Packaging

kxi.package.load_file does not work properly for Python files. For a Python module, use import instead.

Query Tab

There is presently no way to select a subset of the available columns during the getdata phase in the Query tab of the UI. Conversely, the getData API allows this in the agg parameter where you can pass in the list of columns you want with no aggregation function.

UI

Changing the system log level in the UI does not get actioned.
On Mozilla Firefox, when clicking Deploy in a Database screen, the scroll wheels do not render properly in the pop-up.

Views

When displaying streaming data in Views and using the Map component, the map may take some time before the streaming data is displayed. A page reload may be required.
When working with a View then temporarily navigating away to a different tab, any changes made will be saved despite the user not requesting this. If multiple users are editing the same view, any time one switches away from the tab (even if they haven’t made any changes), the current version of their view will be saved, which may not include changes made other users. To resolve this, use the Undo action to return the last desired saved state.
A popup may be shown indicating there may be unsaved changes when no changes have been made.

Scratchpads

Certain Python expressions cause internal errors in Python scratchpads:

lambda x: x * 2
memoryview(bytearray("myTest", encoding='utf-8'))
{1,2,3}

Reliable Transport

RT clients output certain logs to a folder determined by theKXI_C_SDK_APP_LOG_PATH environment variable ($KXI_C_SDK_APP_LOG_PATH/kxi_c_sdk_logs/rt_helper.VMC7AW/rt_helper.log). This variable uses a default location of /tmp. When there are recurring errors, these logs can fill the /tmp directory which is only 5MB in size. To overcome this issue, you can change this environment variable. See Environment variables for steps on how to change this.

Deprecation

The CLI kxi assembly command is being deprecated in 1.9.0 in favor of kxi package and may be removed from future releases. See kdb Insights CLI Reference for more information on the kxi assembly command.
prtnEndCB and reloadCB callback functions are being deprecated in 1.9.0 and may be removed from future releases.
RT maxLogSize is no longer used in configuration and will be ignored from older configurations. RT will now utilize all disk available to it.

1.8.2

Release Date

2024-01-23

Fixes

Expand to see the full detail of resolved issues here

Database

An issue has been resolved in relation to Batch Ingest. Previously, some data in the Historical Database (HDB) was persisted to disk but it was unqueryable. This has been fixed, and all data ingested into the HDB is now queryable.

Third-party Dependencies

kdb Insights Enterprise 1.8.2 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.8.2 release of the standalone infrastructure installation scripts.

Artifacts

Type	Location
Enterprise	insights-1.8.2.tgz
Operator	kxi-operator-1.8.1.tgz
CLI	kxicli-1.8.0-py3-none-any.whl
C interface	kxi-c-sdk 1.8.0
RT Python interface	kxi-rtpy-1.8.0
Java interface	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.8.0.tgz

1.8.1

Release Date

2024-01-16

Improvements

Encryption of data in transit with OpenShift

The encryption of data in transit is now supported when deploying kdb Insights Enterprise using OpenShift.
Stream Processor CSV Decoder Improvements

The Python CSV decoder now supports the first header option. This should be used when only the first batch of your dataset contains a header in the CSV file. See Decoders for information on the CSV decoder configuration.

Fixes

Expand to see the full detail of resolved issues here

Packaging

Resolved an issue where .kxignore files were incorrectly read as bytes instead of strings when compressed on Ubuntu for Windows.

Stream Processor

Stream Processor Beta features configuration has been fixed. When deploying the SP Helm chart, setting betaFeatures: true wasn't correctly enabling Beta features.
A fix has been introduced in the UI to ensure changes to the position of an SP node in a pipeline are now correctly saved.
When creating a pipeline in the UI, the Stream Processor S3 writer now allows for proper writing to an object storage bucket. Previously, this could error with Some nodes have errors, fix them before trying to deploy this pipeline.
Previously, the CSV decoder could drop messages when paired with a file reader (Amazon S3, Google Storage, Azure Storage) where chunking was enabled and the CSV decoder header was set to always. This issue has been resolved. See Decoders for information on the CSV decoder configuration.
SP Cloud Readers no longer error when there are no files matching the expected pattern. Previously, a cloud reader using glob pattern matching would prevent the pipeline from starting if no files matched. In particular, this would occur when trying to use the file watcher feature and no files matched as of the pipeline starting.
The delimiter select field in the Stream Processor CSV node has been updated to allow any delimiter to be used. Previously, there was a restriction on the number of characters that could be submitted, preventing entry of \t or any other delimiter.
Resolved the problem of all window nodes triggering their jobs at the same time, despite their configuration settings. This would occur in setups with pipelines with multiple window nodes, or separate pipelines that read and write to the same RT stream. See Windows nodes for more details.

Security

Various CVEs were remediated as part of this release.

Third-party Dependencies

kdb Insights Enterprise 1.8.1 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.8.1 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.8.1.tgz
Operator	kxi-operator-1.8.1.tgz
CLI	kxicli-1.8.0-py3-none-any.whl
C interface	kxi-c-sdk 1.8.0
RT Python interface	kxi-rtpy-1.8.0
Java interface	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.8.0.tgz

1.8.0

Release Date

2023-12-11

New Features

Data Encryption

Data in transit within kdb Insights Enterprise can now be encrypted, providing enhanced security. This feature meets compliance and regulatory requirements and protects against cyber-security attacks. For further details on how to enable encryption via the CLI, see Encryption of data in transit.
OpenShift Container Platform (OCP) Support

kdb Insights Enterprise can now be deployed on premise using the OpenShift Container Platform (OCP). This expands on the deployment topology options of kdb Insights Enterprise, and enables customers to take advantage of its capabilities without having to run in a public cloud. For further details, see OpenShift infrastructure prerequisites.

Note

See here for details on an issue with Openshift upgrades.
Azure and GCP Parquet Reader Support

The kdb Insights Parquet Reader functionality has been extended to support Azure and GCP object storage. Users can now read, decode, and watch Parquet files stored on Azure and GCP, as well as AWS. These enhancements are available in the UI, the q API and the Python API.
HTTP Publishing

kdb Insights Enterprise has introduced a REST interface to publish data over HTTP. This expands the interoperability of kdb Insights Enterprise for ingesting data. Note that file size is limited to 10MB per publish. See Upload via HTTP for more details.
Python Interface to Reliable Transport

A new Python interface allows you to publish data to Reliable Transport from your own Python application in Docker, or on Linux or Windows. See here for details.
New Database Preview API

You can now query a sample of your data from the database to understand your data/schema, construct more complex queries, or build visuals on the full data set. See here for details.

Improvements

Autoscaling Service Gateways

The Service Gateway and Aggregator can now autoscale if CPU exceeds your configured threshold. The default is to trigger scaling when CPU exceeds 80% for 5 seconds, with a default stabilization period of 300 seconds. All of this is configurable in your values file. See here for details.
CLI Enhancements

The kdb Insights Enterprise CLI contains a number of security and user experience improvements added:
- Browser-based authentication for users, and the ability to authenticate even without a browser
- Ability to authenticate once and still straddle administrator and developer views
- Token caching & refreshing to support the authentication methods
- Use of profiles to allow for concurrent sessions in different environments
See CLI Authentication for details.
UI Enhancements
- When editing a pipeline node, you are now reminded to apply your current changes before moving away to perform other operations. This avoids the potential loss of changes when clicking on to another node.
- Ability to view the rolled-up status of a pipeline group when there are multiple pipelines/replicas.
- Undo and Redo buttons have been added to the Pipeline editor.
- Pipeline settings have been moved into a new ‘Settings’ tab on the Pipeline page. See here for details.
- For tables marked in a foreign key relationship, further validation has been added upon startup to check that these exist and fail fast if any are missing.
- The autocomplete widget in code editors now only appears when manually triggered with Ctrl + Space.
Stream Processor
- Cloud file readers in the Stream Processor now support watching for new files. You can now provide a pattern expression to search for file names. When watching is enabled, the file reader will poll for new files that match the expression. For full details on file watching, see the Amazon S3 reader, the Azure Blob storage reader and the Google Cloud storage reader.
- You can now deploy replicas of pipelines. When deploying, select the “Advanced” option to configure a “Replicas” input field. The environment will deploy the configured number of replicated pipelines identical in configuration with a different ID. See here for details.

Fixes

Expand to see the full detail of resolved issues here

Database

Improved startup performance of HDB DAPs when loading late data from IDB tier mount.
Storage Manager resiliency improvements have been made to improve how SM cleans up aborted transactions after a failure event.
Improved validation for schemas in the UI. Rules now validate against invalid configuration combinations with attributes, sorts and reserved words.
Improved failure recovery for object storage in Storage Manager. This resolves a case where EOI would not complete and would report SM Flushing filesystem with [directory] errors.

Reliable Transport

Logging for LEADER/FOLLOWER updates have been changed to DEBUG level to reduce log noise.

Stream Processor

Previously, it was possible that FINISHED SP pipelines using multiple partitioned workers could spawn extra zombie workers. This has been resolved.
Pipeline quick tests no longer fail if they contained a kdb Insights Stream or kdb Insights Database reader or writer.
Previously it was not possible to maintain state of any object that contained a kdb guid type. This is now fixed. However, if a rollback to an earlier version is required after an upgrade, a rollback step should be run so that on-disk checkpoints can be converted to a format compatible with earlier versions.
Fixed an issue where the csv decoder, with option header=first, and paired with an unbounded streaming reader, was incorrectly dropping the first record in every stream batch.
Fixed a determinism gap when reading objects from cloud storage using a glob pattern. In a recovery scenario, it was possible to receive duplicate data.
Fixed a bug where unpopulated column of type 0h (i.e. list) was not getting filled in with blank entries correctly.
If a pipeline is deployed as part of an assembly, .qsp.write.toDatabase now defaults to writing to that assembly. This means the assembly name can be omitted when calling .qsp.write.toDatabase, for example, .qsp.write.toDatabase[trade; .qsp.use`directWrite!(::;1b)].
Fixed a bug where in a single path pipeline, the existence of more than one emitter resulted in that same number of combine nodes being added.

Upgrades

Resolved a previously-published known issue where upgrades could occasionally fail with an error when running kxi install upgrade. This occurred due to a race condition in the 1.7.0 version of the CLI when trying to replace the CRDs.

File Watchers

When using the file/cloud readers in a non-file watching scenario, files that were not getting chunked did not have their state checkpointed. This meant that during a recovery scenario, duplicate data would be generated for those files.
An issue has been resolved that could cause the file watcher to lose data at the boundaries when chunking was set to auto.

Security

Various CVEs were remediated as part of this release.

User Interface

Resolved the issue where the resizing of a browser window threw the error Error: Cannot read properties of undefined (reading 'getBoundingClientRect') or this.getElements()[0] is undefined.
Resolved an issue where certain Python libraries such as spacy, tensorflow, keras did not work in the Scratchpad.
Postgres and SqlServer nodes no longer incorrectly have a password field.
Resolved the issue where the Views page in the UI occasionally froze when performing certain operations using the map.
The instances where the error "Failed to compare saved assembly [assembly name] to deployed version" were erroneously triggered are now resolved.
Resolved the issue where resizing your browser window throws "Error: Cannot read properties of undefined (reading 'getBoundingClientRect') or this.getElements()[0] is undefined".
Duplicate filters are no longer being reported after creating then saving a filter.
Navigating to a Dashboard/Screen from Actions is now working in Views. Previously, you could receive the error "Error: Cannot read properties of undefined (reading 'navigate')" or be redirected to the homepage.
Creating a View with a Datagrid, copy it to a second View and pasting, and then returning to your original View would cause your changes to be lost. This has now been resolved.
Fixed the incorrect JSON file format that was causing issues when trying to download pipeline logs from the UI.

Third-party Dependencies

kdb Insights Enterprise 1.8.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.8.0 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.8.0.tgz
Operator	kxi-operator-1.8.0.tgz
CLI	kxicli-1.8.0-py3-none-any.whl
C interface	kxi-c-sdk 1.8.0
RT Python interface	kxi-rtpy-1.8.0
Java interface	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.8.0.tgz

Upgrade notes

Encryption of data in transit{:target="blank"} is turned on by default for fresh installs. When you upgrade _kdb Insights Enterprise the CLI gives you the option to turn encryption on or off. To change this setting after installation, perform an upgrade.
When creating a new database through the kdb Insights Enterprise UI, the isSharded schema property is now being defaulted to true. See here for more details on the implications of partitioned tables.

Known Issues

Upgrading kdb Insights Enterprise on Openshift will report that the insights-on-k8s task will fail to install Istio. This can be safely ignored. Please follow the instructions to install Istio as a prerequisite.
Using a Stream Processor Amazon S3 reader may fail when running on EKS. To workaround this, please set the tenant field to kxinsights.
If you have has an existing kxi-license secret created from a kc.lic or k4.lic, kxi install setup will re-use this secret, but will not set the appropriate values in the resulting values.yaml. See the available license types here.
Occasionally installs and upgrades may fail with an error when running kxi install. This can occur due to timeouts with the helm upgrade, giving errors such as the below:
```
    timed out waiting for the condition
    context deadline exceeded
```
Re-run the install or upgrade to resolve this issue.
With entitlements enabled, the Query tab in the UI will not return data as there are issues with resolving the correct database name when checking entitlements. In order to workaround this issue, you should create a second entitlement for each database / assembly.

For example if you create and entitle an assembly called dfx-assembly, you should also create an identical entitlement called dfx-assembly-qe. This -qe entitlement should be kept in-sync when updating the main one.

This restriction will be removed in a future release.
qsql requests, made either through the Q tab (Query environment) in the UI or via REST to /qsql, do not work with entitlements. The .kxi.qsql API should be disabled if using entitlements. This is disabled by default in kdb Insights Enterprise. See here for more details.

1.7.4

Release Date

2024-01-04

Fixes

Expand to see the full detail of resolved issues here

Database

Improved failure recovery for object storage in Storage Manager. This resolves a case where EOI would not complete and would report SM Flushing filesystem with [directory] errors.
Resolved a bug in which EOIs were not completing, and NoSuchKey errors were occurring after moving data to object storage.
Improvements have been made to the refresh token logic for AWS deployments. Previously, token refresh logic was erroneous if the session was not requesting object storage bucket data. The refresh rotated every 30 minutes instead of at the session expiry time. Session expiry time is configured with EC2 instance metadata, or by KX_KURL_AWS_TOKEN_DURATION when using Amazon STS.

User Interface

Resolved an issue where certain Python libraries such as spacy, tensorflow, keras did not work in the Scratchpad.
In the Query tab, duplicate filters are no longer being reported after creating then saving a filter.

Security

Various CVEs have been remediated as part of this release.

Third-party dependencies

kdb Insights Enterprise 1.7.4 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.7.4 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.7.4.tgz
Operator	kxi-operator-1.7.2.tgz
CLI	kxicli-1.7.2-py3-none-any.whl
C interface	kxi-c-sdk 1.7.0
Java interface	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.7.0.tgz

1.7.3

Release Date

2023-11-01

Fixes

Expand to see the full detail of resolved issues here

Database

Optimizations made to the getData API to improve query performance when the prtnCol is sorted.

Security

Various CVEs have been remediated as part of this release.

Third-party dependencies

kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.7.3.tgz
Operator	kxi-operator-1.7.2.tgz
CLI	kxicli-1.7.2-py3-none-any.whl
C interface	kxi-c-sdk 1.7.0
Java interface	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.7.0.tgz

1.7.2

Release Date

2023-10-03

Improvements

SQL Queries
- SQL queries will now use less memory if late data is enabled and you are querying data for the current day.

Fixes

Expand to see the full detail of resolved issues here

Database

Resolved an issue where data for dates modified after the retention period was not getting moved from the first to the secondary HDB tier.
Improved the startup time for HDB DAPs, allowing for faster availability of processes that serve queries.

Stream Processor

The Apply Schema transform node now works with NULLs in string types.

Third-party dependencies

kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.7.2.tgz
Operator	kxi-operator-1.7.0.tgz
CLI	kxicli-1.7.1-py3-none-any.whl
C interface	kxi-c-sdk 1.7.0
Java interface	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.7.0.tgz

1.7.1

Release Date

2023-09-19

Fixes

Expand to see the full detail of resolved issues here

Database

Resolved an issue with the Storage Manager not rolling back aborted transactions if an EOD failed.
Deleting data from the HDB using a filter option now correctly performs the delete. Previously, this returned an error claiming that the column(s) being filtered didn’t exist.
EOIs are no longer delayed after an EOI recovery event. Previously, it was possible for EOIs to become held up if the process was terminated then subsequently recovered. The EOIs would still complete successfully, but with a delay.

Security

Resolved CVE-2023-37920 Certifi's removal of e-Tugra root certificate, by upgrading the spaCy dependency.

Stream Processor

Attempts to horizontally scale pipelines using the direct write feature will now explicitly error since this is not currently supported.

Deployments & Upgrades

Resolved the known issue in 1.7.0 in which a race condition could cause an upgrade to fail with an error when running kxi install upgrade.

Third-party dependencies

kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.7.1.tgz
Operator	kxi-operator-1.7.0.tgz
CLI	kxicli-1.7.1-py3-none-any.whl
C interface	kxi-c-sdk 1.7.0
Java interface	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.7.0.tgz

1.7.0

Release Date

2023-08-25

New Features

End-of-day writedown memory controls

The Storage Manager (SM) process now supports restricting memory consumption for data writedown, giving the opportunity for cost savings. Reducing the RAM allocated to SM may cause EOD to run longer than if RAM was unrestricted.

For guidance on tuning your RAM setup, see Performance best practices and the recommended system sizes for Azure here.
One-node RT

kdb Insights Enterprise now allows you to optionally deploy an RT Stream as a one-node cluster instead of a three-node cluster when deploying via the kdb Insights CLI.
- In a three-node cluster, RT provides fault tolerance and high availability (HA).
- In a one-node cluster, RT provides fault tolerance, but the cluster is not highly available. This configuration saves resources and reduces costs for workload use cases that do not need to be highly available. In the event that the single RT node goes down, the publisher then continues to write to its local RT log file. When the RT node restarts, the publisher retrieves any data that is present on the publisher node that it has not yet received.
See here for more details on RT architecture.
C Subscriber support in Reliable Transport

The C language interface for RT is now able to subscribe to the data on an RT stream. See here for more details.
Backfill a new column with derived data

When adding a new column to your schema, you can now define a custom function that will backfill the new column with derived data.
Publishing derived data from custom APIs

Custom APIs that generate derived data can now stream and/or persist their output.
Appending data on batch ingest

When ingesting data using the direct write feature, you now have the option to append data rather than replace it. See overwrite option for more details.
Data Entitlements for querying data Beta

When querying via REST or IPC, kdb Insights Enterprise now provides data entitlement restrictions for limiting user access to the data contained within specific deployed databases.
Parquet file reader Beta

Ingesting Parquet files from local storage or from an AWS S3 bucket using the q reader API, Python reader API and the UI is now enabled. This feature will be enabled for Azure and GCP and include a file watcher with file pattern matching in a future release.

Improvements

End-of-day writedown performance improvements

The Storage Manager (SM) process has been optimized to significantly improve its performance - up to 50-70% reduction in EOD duration. This improved EOD performance can reduce the memory allocated or the time taken for writedown, depending on your workloads and configuration.

For guidance and best practices on how to configure your schema and system to take advantage of these optimizations, review Performance best practices and the recommended system sizes for Azure here.
User interface
- The database schema editor has been completely redesigned with a new responsive table grid for entering column metadata details. The editor uses keyboard driven actions for efficient data entry. Backed by HTML canvas rendering, hundreds of columns can be added with ease.
- The Database configuration now supports different storage classes for the IDB and HDB (for example, FSx for Lustre). There is now a dropdown under Advanced Writedown settings for the File System that changes the mount storage class. See here for details.
- The code editors in the Pipeline Editor can now be used to evaluate q and Python code. See Executable Editors in the Building a Pipeline section for more information.
- Virtual queries are now available in Views. See Virtual data sources for more information.
- The ‘Database Wizard’ has been removed from the UI. Databases can now be created and configured using the unified database configuration screens.
- Query results in the UI are now shown using the timezone as returned by the query APIs for both Console and Table views. Previously, Table view would apply the user’s timezone which may have been different.
Readers
- When using cloud and file readers, you can now use filename pattern matching to ingest multiple files using a single reader. This expands the ability for you to build up ingest workloads that pull in multiple files. For example, using an S3 reader, you could read all CSV files in all folders within a bucket using the pattern:
```
s3://path/to/bucket/**/*.csv
```
- When using cloud and file readers you can now use a file watcher which is usually paired with filename pattern matching described above. The reader will continually watch for new files matching the file path(s) provided. See here for details on:
  - q APIs: .qsp.read.fromGoogleStorage, .qsp.read.fromAzureStorage and .qsp.read.fromAmazonS3.
  - Python APIs: .qsp.read.fromGoogleStorage, from_azure_storage and from_amazon_s3.
  - UI: Google Cloud Storage, Microsoft Azure Storage and Amazon S3.
  Not to be used with the Write Direct to HDB option
  
  The Watcher options should not be used when also choosing the Write Direct to HDB option in the Database Writer in the UI, the q or Python APIs.
- Advanced configuration options for the Kafka Reader node in the Pipeline Builder UI allow more flexible options for security protocol configuration or changes to fetch intervals, as seen when subscribing to an Azure Event Hub Kafka connector.
Reliable Transport
- Users can now launch KXI feeds from outside of their Kubernetes cluster and send data to their kdb Insights Enterprise deploy.
Logging
- RT Stream logs are now included in Database logs on the Diagnostics page, and they are also included in the Database Issues counts shown on the Database Overview page.
- When viewing a deployed pipeline, you can now click on an icon to jump straight to the pipeline logs.
Terraform Scripts
- The default node types for each cloud provider have been changed to ensure that the default node group uses memory-optimised instances, and the node group for rook-ceph has storage-optimised instances:
```
    AWS:

    default node pool: "r5.xlarge"

    rook-ceph node pool: "i3.2xlarge"

    Azure:

    default node pool: "Standard_E4_v3"

    rook-ceph node pool:”Standard_L8s_v2"

    GCP:

    default node pool: "n1-highmem-4"

    rook-ceph node pool:”n2-standard-8"
```
- The deployment of the nginx controller has been changed to be a Daemonset which will ensure consistent availability across all nodes.
- The standalone installation using the terraform client scripts now enables cloud native system metrics on GKE clusters.

Fixes

Expand to see the full detail of resolved issues here

Database - When using the Filter By Label option in the Query page, you can now use more than one label for the filter. In previous releases, additional labels beyond the first were not taken into account when the query was executed.

Fixed an issue where schema type conversions from strings to GUIDs would fail and cause a type error.
Fixed an issue on certain NFS-based system (namely Lustre) where the ‘current’ database pointer could get converted to a directory resulting in an unexpected error.

EISDIR: /data/idb/current.new -> /data/idb/current

An error has been fixed that would occur when updating an object storage tier that used an inventory file after a data migration.
Fixed an issue when querying with .kxi.getData using a groupBy operation on a foreign key column where the name of the foreign key column would be thrown and the query would fail to complete.
Fixed an issue where renaming columns in the schema UI would incorrectly delete the underlying column and create a new one with null data. Column renames in the UI now correctly preserve the column’s data.
A new warning has been added to the Resource Coordinator if no Aggregators connect to it. After a wait period, if an Aggregator continues not to connect, this will be escalated to an error to indicate that the system is not ready for queries.
Stream resources now correctly show the amount of resources to be used when an additional stream was added. Previously the stream resources were only showing the compute settings for a single stream instead of adding both together.
When the kdb Insights Database was configured with an object storage tier but an inventory file was not provided, the Storage Manager would error on startup with invalid object storage URI error. This has been changed to not attempt to create an inventory file if no configuration is explicitly provided.
Resolved an issue when attempting to migrate HDB data to Azure Blob Storage. Previously, this would fail with a signature error.
Fixed a potential race condition between NFS replication and DAP reload signals. It was previously possible that a DAP may reload before NFS replicated data which could result in data gaps. A new sequencing mechanism has been added to ensure DAPs do not reload until replication is complete.
Values in a symbol column will now be properly available immediately after a batch ingest. Previously, these would be missing until the next EOI.
Renaming a sort column now longer errors.

Stream Processor

q code in a pipeline’s Global Code field will now be executed with the same semantics as loading a .q file. This fixes issues around expressions incorrectly being concatenated, the handling of comments, and expressions starting with .
The return of UDFs when generating q objects is no longer incorrectly wrapped as foreign objects. A user no longer needs to use a null python function following the UDF to unwrap it to the correct type.

Command Line Interface - Reinstalls and rollbacks no longer fail due to password history issues with the initUser.

Third-party dependencies

kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.7.0.tgz
Operator	kxi-operator-1.7.0.tgz
CLI	kxicli-1.7.0-py3-none-any.whl
C interface	kxi-c-sdk 1.7.0
Java interface	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.7.0.tgz

Upgrade notes

Occasionally upgrades may fail with an error when running kxi install upgrade. This occurs due to a race condition in the 1.7.0 of the CLI when trying to replace the CRDs. The upgrade will stall and timeout at this point.
```
..
Replacing CRD assemblies.insights.kx.com
Replacing CRD assemblyresources.insights.kx.com
Error: Timed out waiting for CRD assemblyresources.insights.kx.com to be deleted
```
Re-running the upgrade should workaround the issue. This will be fixed in the next patch release.

Known Issues

With entitlements enabled, the Query tab in the UI will not return data as there are issues with resolving the correct database name when checking entitlements. In order to workaround this issue, you should create a second entitlement for each database / assembly.

For example if you create and entitle an assembly called dfx-assembly, you should also create an identical entitlement called dfx-assembly-qe. This -qe entitlement should be kept in-sync when updating the main one.

This restriction will be removed in a future release.
Previously cloud readers in a failure recovery scenario would begin reading file(s) from the start, even if they had read them prior to crashing. This would have caused duplicate data if using a stream or database writer. As of 1.7 this is no longer the case. One exception case remains, when using a glob pattern match in a non watching scenario. If the pipeline fails, recovers and matches a new set of files (because the underlying files in the object storage have changed) we may get duplicate data.
qsql requests, made either through the Q tab (query environment) in the UI or via REST to /qsql, do not work with entitlements. The .kxi.qsql API should be disabled if using entitlements.
Occasionally installs and upgrades may fail with an error when running kxi install. This can occur due to timeouts with the helm upgrade, giving errors such as the below:
```
    timed out waiting for the condition
    context deadline exceeded
```
Re-run the install or upgrade to resolve this issue.

Backward Compatibility

If you are upgrading from a version earlier than 1.6.3 you must also read each of the relevant release notes to ensure you know all the changes that will affect the upgrade. Refer to the previous release page for older releases.

1.6.4

Release Date

2023-11-28

kdb Insights Core Version

This version of kdb Insights Enterprise runs kdb Insights Core 4.0.5.

Fixes

Expand to see the full detail of resolved issues here

Database

Improved failure recovery for object storage in Storage Manager. This resolves a case where EOI would not complete and would report SM Flushing filesystem with [directory] errors.
Resolved a bug in which EOIs were not completing, and NoSuchKey errors were occurring after moving data to object storage.
Improvements have been made to the refresh token logic for AWS deployments. Previously, token refresh logic was erroneous if the session was not requesting object storage bucket data. The refresh rotated every 30 minutes instead of at the session expiry time. Session expiry time is configured with EC2 instance metadata, or by KX_KURL_AWS_TOKEN_DURATION when using Amazon STS.
Added a fix for the GCP system command expiry timer being discarded/ignored, which resulted in tokens not refreshing at all using gcloud auth.

Security

Various CVEs were remediated as part of this release.

Artifacts

type	location
Enterprise	insights-1.6.4.tgz
Operator	kxi-operator-1.6.1.tgz
CLI	kxicli-1.6.1-py3-none-any.whl
C SDK	kxi-c-sdk 1.6.0
Java SDK	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.6.0
Infrastructure	kxi-terraform-1.6.1.tgz

1.6.3

Release Date

2023-07-25

Improvements

A new quick test deploy option is available for pipelines for faster iteration during development. This deploys a pipeline in test mode to the user’s scratchpad process and is significantly faster than a full test. See quick test deploy for more information.

Fixes

Expand to see the full detail of resolved issues here

Database

Fixed a potential race condition between NFS replication and DAP reload signals. It was previously possible that a DAP may reload before NFS replicated data, which could result in data gaps. A new sequencing mechanism has been added to ensure DAPs do not reload until replication is complete.
Fixed an issue where converting a column from a string to a symbol would throw a type error.
An error has been fixed that would occur when updating an object storage tier that used an inventory file after a data migration.
A new warning has been added to the Resource Coordinator if no Aggregators connect to it. After a wait period, if an Aggregator continues not to connect, this will be escalated to an error to indicate that the system is not ready for queries.
REST queries could have input parameters converted to the wrong type when multiple type options were possible. REST APIs were converting to the maximum possible type instead of using the correct default type.
Database names in diagnostics now correctly correspond to the name of the assembly that the database maps to.

User Interface

Any change made on the Database UI tab will now mark the database as unsaved, and warn the user if trying to close the tab with any unsaved changes.
The View Errors button is now renamed to View Database Errors and points to the database error logs. To view Pipeline Errors, click on the Open Pipeline Errors icon next to each pipeline error.
Fixed an issue where renaming columns in the schema UI would incorrectly delete the underlying column and create a new one with null data. Column renames in the UI now correctly preserve the column’s data.

Artifacts

type	location
Enterprise	insights-1.6.3.tgz
Operator	kxi-operator-1.6.1.tgz
CLI	kxicli-1.6.1-py3-none-any.whl
C SDK	kxi-c-sdk 1.6.0
Java SDK	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.6.0
Infrastructure	kxi-terraform-1.6.1.tgz

1.6.2

Release Date

2023-07-03

New Features

A file watcher for polling cloud storage is now supported in the Stream Processor. The file watcher can monitor any Azure, AWS, or GCP object storage directory. It polls for new files to ingest into an SP pipeline. See the q reader API for details on how to configure the watch parameter in your SP reader. This feature will be available in the Python API and UI in a future release.
Query resiliency has been improved by adding the ability to control how processes serving queries are shut down. These processes can now be configured to stay up to serve in-progress query requests before shutting down, ensuring users still receive their data and are unimpacted by system operations. Users simply define how long a process should continue to serve requests before ultimately going offline. This is configured by setting .values.service-gateway.aggregator.aggregator.terminationGracePeriodSeconds to an appropriate duration in your values file during installation.
Query scalability has increased with the new ability to independently scale query processes within an assembly, allowing each assembly to have a tailored amount of these processes. Making this configurable and manually scalable lets you better size your assemblies with specific query workloads in mind. See routing configuration for more details. Furthermore, a new routing parameter called scope supports routing query requests based on your target assembly(s).
Reading directly from a Parquet file(s) in local storage is now supported in Stream Processor pipelines, allowing users to take advantage of this efficient common file format. See the q reader API for more details. This feature will be available in the Python API and UI in a future release.
Connect your existing kdb database already in object storage to your kdb Insights Enterprise system and query it.
Pipelines created in the UI can now be exported to be run by the CLI, either for automation or for migrating between environments. From the UI, select the pipeline and click ‘Export’. See Exporting artifacts for more details.

Fixes

Expand to see the full detail of resolved issues here

User Interface

In the kdb Insights Enterprise UI, new validation has been added for table names within schemas to ensure that they do not contain invalid characters. Previously, invalid characters could be included in tables names and the database would error after it was deployed.
In the kdb Insights Enterprise UI, you can now allow an internal stream to read data from an external source directly to a database. See stream configuration for more details.
When creating a new database or renaming an existing database, you are prevented from entering a database name longer than 24 characters.

Artifacts

type	location
Enterprise	insights-1.6.2.tgz
Operator	kxi-operator-1.6.1.tgz
CLI	kxicli-1.6.1-py3-none-any.whl
C SDK	kxi-c-sdk 1.6.0
Java SDK	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.6.0
Infrastructure	kxi-terraform-1.6.0.tgz

1.6.1

Release Date

2023-06-22

Improvements

In the kdb Insights Database, a new routing parameter called scope has been introduced for routing query requests based on assembly. This parameter allows the user to specifically target individual assemblies for their query.

Artifacts

type	location
Enterprise	insights-1.6.1.tgz
Operator	kxi-operator-1.6.0.tgz
CLI	kxicli-1.6.0-py3-none-any.whl
C SDK	kxi-c-sdk 1.6.0
Java SDK	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.6.0
Infrastructure	kxi-terraform-1.6.0.tgz

1.6.0

Release Date

2023-06-20

Improvements

Support for air-gapped environments has been improved and both kxi install run and kxi install upgrade support installing packaged charts (chart tgz’s) by passing the packaged chart name as an argument. If you use the default value of kx-insights for the chart repo, you can install it as follows:
```
kxi install run --version 1.6.0 --filepath values.yaml insights-1.6.0.tgz
```
This argument also supports chart references, for example:
```
kxi install run --version 1.6.0 --filepath values.yaml <CHART_REPO>/insights
```
The kdb Insights Database can now work with or without an inventory file provided for an object storage tier. Using an inventory file for your object storage tier is still the recommended configuration, as this leverages caching and greatly improves performance when querying object storage data.
The kdb Insights Enterprise UI has an improved left hand menu. It can be minimized, and it is displayed alongside the Pipeline Operators list when the pipeline editor is active.
The kdb Insights Database now validates that primary keys for a partitioned table include the partition column. Previously, setting a primary key on a partitioned table that was not the partition column would error when queried.
Query timeout messages now indicate a reason for the timeout. Previously, query timeouts gave no reason, or said “timed out for unspecified reasons”. See queueing timeouts for more details.
Greater stability through the use of Kubernetes discovery between the Service Gateway and database processes.
Client-side pivot queries are now available in Views, with optional server-side computation for kdb+/q.
When calling kxi install delete the kdb Insights CLI now backs up assemblies before tearing them down.

Fixes

Expand to see the full detail of resolved issues here

Security

Various CVEs have been remediated as part of this release.

Database

Resiliency improvements have been made to ensure the kdb Insights Database can recover from multiple days without end-of-day processing having completed. Previously, if more than 2 EODs were missed, it was possible to get a misaligned error on recovery.
Validation has been added for assemblies to ensure that tiers' scheduled alignment time (snap value) are provided as the kdb+ time type (19h). Previously users could enter a timespan which would silently be ignored. Entering a timespan will now error.

Reliable Transport

Kubernetes service account tokens are now automatically refreshed from disk. In previous versions these were not refreshed, causing failing requests to the Kubernetes control plane and pods had to be restarted.

Deprecation

CLI Install Commands

The following three arguments have been deprecated from kxi install setup: chart-repo-name, chart-repo-username and chart-repo-url.

To prevent breaking changes the arguments will still work, but they may be removed in a future version. Adding a chart repository is now decoupled from kxi commands. The suggested approach when calling kxi install setup is to add the chart repository directly using helm repo add and to reference that repository by passing the chart reference as an argument.

If you use the default value of kx-insights for the chart repo, you can install it as follows:

kxi install run --version 1.6.0 --filepath values.yaml

If you have used a non-default chart repo value then you can install with:

kxi install run --version 1.6.0 --filepath values <CHART_REPO>/insights

CLI Lifecycle Commands

The following commands have been removed. Please use the equivalents below to manage assemblies.

removed	supported command
kxi azure assembly backup	kxi assembly backup
kxi azure assembly delete	kxi assembly teardown
kxi azure assembly restore	kxi assembly deploy

Third-party dependencies

kdb Insights Enterprise 1.6.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.6.0 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.6.0.tgz
Operator	kxi-operator-1.6.0.tgz
CLI	kxicli-1.6.0-py3-none-any.whl
C SDK	kxi-c-sdk 1.6.0
Java SDK	kxi-java-sdk 1.6.0
RT Bridge	rt-bridge 1.6.0
Infrastructure	kxi-terraform-1.6.0.tgz

Known Issues

QueriesLicensingCLIPipelinesRollbackReinstall

In 1.6.0 only, any code evaluated in the Python editor will throw a type error. To fix the Python editor for this issue you can run the following in a q editor:

\d .com_kx_edi
python.run: {
    z: python.i.removeExtraIndents z;
    a: python.i.run[z; y];
    $[x;$[y;python.i.truncate (),a;a];$[y;"";::]]
    }

When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:
- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit super back to sp
- Press the down arrow to go to the next line

On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.
A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below
```
klic license renew $LICENSE_UUID --version=latest

env QLIC=$(pwd) klic license fetch $LICENSE_UUID
```
Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.
```
printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml

kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml
```

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. Therefore, there is a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

A known issue exists within the kxi install rollback command when rolling back to prior versions. When the target version was installed with keycloak.initUser enabled. It is expected the keycloak-config-cli job will fail. This is due to the rollback trying to revert that user's password and violating the password history policy.

The rollback failure will manifest as the failure of the keycloak-config-cli job. This will create job pods as it fails and retries. In order to proceed with the rollback it's recommended to delete kubectl delete job ${RELEASE_NAME}-keycloak-config-cli.

A known issue exists during a reinstall using the CLI where the keycloak.initUser is enabled. If the user's password has changed, the reinstall will try to revert it back and be rejected by the password history policy.

This can be avoided by disabling the user import on reinstall using --import-users False as below:

kxi install run --import-users False ..

Backward Compatibility

If performing a rollback from 1.6.0 to 1.5.x, a rollback chart is required. See Upgrading for more details.

1.5.3

Release Date

2023-06-19

Improvements

Resiliency improvements have been made to ensure the kdb Insights Database can recover from multiple days without end-of-day processing having completed. Previously, if more than 2 EODs were missed, it was possible to get a misaligned error on recovery.
The kdb Insights Database can now work with or without an inventory file provided for an object storage tier. Using an inventory file for your object storage tier is still the recommended configuration, as this leverages caching and greatly improves performance when querying object storage data.
In cases where no data transformation is required, the kdb Insights Enterprise UI now allows Streams to be configured to write directly to the database. This reduces the amount of resources required for data ingestion and lowers operating costs.

Fixes

Expand to see the full detail of resolved issues here

Security

Various CVEs have been remediated as part of this release.

User Interface

When creating a new database or renaming an existing database, you are now prevented from entering a database name longer than 24 characters. Previously, there was no restriction which could lead to name truncation.
The Database Schema Code View page now accepts all schema configuration options. Previously, it would not accept blockSize or isSharded.

Reliable Transport

Kubernetes service account tokens are now automatically refreshed from disk. In previous versions these were not refreshed, causing failing requests to the Kubernetes control plane and a need to restart your pods to recover.

Artifacts

type	location
Enterprise	insights-1.5.3.tgz
Operator	kxi-operator-1.5.3.tgz
CLI	kxicli-1.5.0-py3-none-any.whl
C SDK	kxi-c-sdk 1.5.0
Java SDK	kxi-java-sdk 1.5.0
RT Bridge	rt-bridge 1.5.0
Infrastructure	kxi-terraform-1.5.1.tgz

1.5.2

Release Date

2023-05-31

Improvements

In the kdb Insights Enterprise UI, when filtering queries in the Query page, distinct is now a supported aggregation type.
The getData API can now perform aggregations on foreign key columns.

Fixes

Expand to see the full detail of resolved issues here

Kubernetes service account tokens are now automatically refreshed from disk. In previous versions these were not refreshed, causing failing requests to the Kubernetes control plane and pods had to be restarted.

Database

When deploying an assembly via the kdb Insights CLI, if ‘size’ is not specified for DAP instances, a default of 3 replicas will once again be applied. In 1.5.1, this default was increased to 4 to provide increased availability, but 3 has been restored as the standard to control costs. Users can configure a greater number of replicas if desired.
In the kdb Insights Enterprise UI, pipelines associated with a database now show in the status table. Previously, only the first pipeline would be shown.
In the kdb Insights Enterprise UI, the database schema overview now updates when changes are made to the database’s schema. Previously, this would only show the schema state when first viewed.
In the kdb Insights Enterprise UI, database documents no longer get marked as modified after a teardown or cleanup.
The database configuration UI now validates that the minimum rollover time is 1 minute. Previously, this could be set to 0.

Stream Processor

Pipelines can be now be safely torn down without checkpoints having been cleared. Previously, if a teardown was followed by a restart, it is possible the pipeline would never run.

Artifacts

type	location
Enterprise	insights-1.5.2.tgz
Operator	kxi-operator-1.5.2.tgz
CLI	kxicli-1.5.0-py3-none-any.whl
C SDK	kxi-c-sdk 1.5.0
Java SDK	kxi-java-sdk 1.5.0
RT Bridge	rt-bridge 1.5.0
Infrastructure	kxi-terraform-1.5.1.tgz

1.5.0

Release Date

2023-05-09

New Features

UI

The kdb Insights Enterprise UI has a new and improved look and user experience for viewing the details of your database(s). All relevant details about a database are now grouped together in a single summary page, giving users an intuitive view into the details and health of its components. Users can quickly see if any errors or warnings are present, and drill down into the details to investigate directly from the page. The details of your schema are conveniently presented for reference as well. To accompany this new interface, the term Assemblies has been simplified to Database in the UI. Assemblies remain the underlying concept tying components together, and are still referenced in documentation. Manual upgrade of old streams and schemas may be required. See upgrading entities in the UI for more details.

Queries

To protect against reserved word and column name conflicts, the way labels are passed to a query has changed. Labels should now be specified either as a dictionary, or with the prefix label_. The old label style is still supported in 1.5, but is deprecated and will be disabled by default in 1.6. See upgrade considerations on labels for more details.
Distinct has been added to the available aggregations in getData for database queries.

Pipelines

Users can now use Python code in all areas of the pipeline UI, allowing for further flexibility in Python user experience with kdb Insights Enterprise. Previously, the addition of Python code to sections of pipeline UI were limited in scope to the function cells of the pipeline configuration forms.
The Stream Processor Python API can now leverage automatic schema conversion using a new function kxi.sp.transform.schema.
Users can now use the Kafka Schema Registry in Python and the UI. See kxi.sp.read.from_kafka for Python, and the Kafka reader.
String manipulation utilities have been added to the UI and the Stream Processor Python and q APIs, allowing users to selectively convert the string data to upper or lowercase representations for the purpose of data cleansing.
Non-UI deployments of Stream Processors now support pipeline replicas, providing enhanced resilience to deployed pipelines. Users can now choose to run their pipeline with replicas for high availability in the Stream Processor. This replaces the previous use of pipeline groups.

Reliable Transport

Reliable Transport now offers subscribers the ability to use dynamic client-side filtering, enabling users to control what parts of the data feed their subscribers need to receive. This can greatly improve ingestion performance by eliminating the need for subscribers to handle discarding of unnecessary data. See Filtering for more information.
RT has introduced logHistory, a new API to assist in analysis and debugging. The API returns metadata about the RT log file history for logs that are no longer available. This includes:
- The log file name.
- The time of the first message added to that log file.
- The position of the first message added to that log file.
See Log File Meta Data for more information.

Command Line Interface

A rollback command has been added to the CLI to simplify rolling back your installed instance of kdb Insights Enterprise after upgrading.
A kxi user subcommand has been added to the CLI to support managing users programmatically. This provides greater flexibility than using the keycloak initUser values at deploy time.

Standalone Installation

Ability to control which components should be enabled (cert-manager,ingress-nginx, sharedfiles storage class). The available components are defined on the configuration file (kxi-terraform.env).

Improvements

Batch ingest now supports ingesting tables that have no symbols. Additionally, extra validation has been added to verify symbol integrity.
Batch ingest now supports sorting of columns.
The kdb Insights Database gateway now supports returning responses that are greater than 2GB for IPC requests. See routing configuration for more details.
Database schemas in the UI can now be modified for assemblies that have been stopped. See schema modification.
Requests that supply a log correlator (logCorr) will now see that correlator in the Service Gateway logs. Previously a random correlator was used in the Service Gateway logs.
Users can now utilize backtick syntax when defining aggregations in Views, e.g. `c1`avg`price.
Pipeline editors can now be resized vertically.
Labels for improved accessibility have been added to the Pipelines screen.
Added significant improvements for accessibility using keyboard navigation and/or screen readers in the UI.
- Users can exit a text area in a code block on the Query page by first pressing "CTRL+M" before pressing the "Tab" key.
- The entity tree menu button can now be opened when hovering your mouse over the entity tree. Arrows can then be used to navigate.
- "Code View" on the Schemas page can now be closed with the "Esc" key.
- The Views and Pipeline pages now allow draggable elements to be inserted/dragged onto the page with the use of your keyboard.

- Kubernetes 1.25 is now supported and the default version for standalone installations.

- Ensure the latest AWS CLI is installed on the docker image used for standalone installations

Fixes

Expand to see the full detail of resolved issues here

Database

Additional historical tiers that used object storage in the UI now allow dashes and spaces in file names. Previously they would error for dashes and spaces.
Migrating tiers to object storage could fail with the eventual error SMDBM Migration discontinued due to failure if multiple threads were used.
New validation has been added to prevent accidentally adding a object storage tier to a database that does not already have one. This would previously error at EOD with a file not found error.
Schema conversions are now more robust on network based file systems (ex. Rook Ceph). Previously, this could result in a EXDEV. OS Reports: Invalid cross-device link error in the Storage Manager.
Changes to disk sort columns in a schema that would be incompatible with the current disk sort will now error with a clear message at deploy time. Previously, this would error with a u-fail error either at an EOI or EOD.
Batch ingest can now gracefully handle scenarios where there is no data provided in the session. Previously, this would result in the following error: Error reporting to licensing server: Value must be a long, leaving the ingest pipeline stuck in a RUNNING state.
When removing a symbol column from a table, the SM should be able to properly handle its removal as well as the sym file.
If an in-memory sort is applied to a column in a schema, it is now sorted on arrival in the RDB. Previously, if data arrived out of order with this set, an s-fail error would be thrown.
In the Storage Manager metrics, the kxi_sm_hdb_date_records metric no longer reports an empty value at startup, which resulted in false alerts about the HDB being offline.
The Storage Manager now double buffers its state table in order to protect against corruption.
Rapid successive queries using qsql, such as those from Views in the kdb Insights Enterprise UI, no longer have the potential to return query results out of order.
The kxi_sg_pending metric would previously be decremented twice when a connection closed, resulting in negative pending values.
The Service Gateways were creating duplicate connections to Resource Coordinators while they were waiting to come online. This was resulting in a large number of redundant connections being opened between the services. In large deployments, this could result in conn errors in the Resource Coordinator.
RDBs and HDBs that require a dependency on the IDB will now error at startup if one is not specified as part of the mount configuration.

User Interface

Previously, a database would show a loading spinner if one of its pipelines had reached a ‘finished’ state while its other components were in a ‘ready' state. This has now been rectified and the database will display a ‘ready’ state in this case.
An issue with pasting into text fields in a pipeline node’s configuration has been fixed. Previously, the node may be been duplicated.

Stream Processor

Pipeline names are now properly displayed in the Running Pipelines section of the Overview page. Previously, these were displaying the pipeline ID and were not clickable for viewing further information.
Pipelines deployed via as part of a Database now properly set Kubernetes secrets. Previously, secrets were only being set properly for standalone pipelines.
The presence of a UTF-8 byte-order mark at the beginning of CSV files will now be automatically removed from data when acted on by the Stream Processor CSV decoder.
Expression readers now return the result of Python lambdas.
Window nodes in the Stream Processor can now accept dictionaries as input batches. Previously, incoming dictionaries would be merged with input batch in the buffer, instead of appending to a buffered list of dictionaries.
The python write.to_database node previously converted assembly names to booleans. This is now resolved, and also fully supports the direct write functionality.
The Stream Processor's Protobuf decoder no longer ignores the value of As List when a pipeline is deployed from the UI. Previously, this value was being overwritten.
The Stream Processors Protocol Buffer Encoder node now properly saves the selected Payload Type.
The Stream Processor Database Reader now supports SQL statement queries. Previously, this node had erroneously suggested the use of qsql statements for querying which was unsupported.
The default onFinish functions for the apply node have been updated to have the correct parameters and examples of flushing a buffer.
When using directWrite functionality in .qsp.write.toDatabase, table names as strings are now supported. Previously, this had to be a symbol.
Copy/pasting nodes within a pipeline document while other pipeline documents are open in the background no longer produces duplicates. Previously, the copy/paste operation was being applied to both the intended pipeline as well as the unintended pipelines, resulting in nodes disappearing due to shared names/ids.
Security vulnerabilities patched in the Stream Processor for the following CVEs: CVE-2022-41910, CVE-2022-41902, CVE-2022-41900, CVE-2022-41880.
Query sessions had previously set error trapping to 1 within the q session, resulting in users being unable to interact with their query session. This behaviour manifested in users being unable to interact with their query sessions until they were manually restarted.

Command Line Interface

With previous versions of the kdb Insights CLI, the kxi install list-versions command would return an outdated list of versions. This has now been resolved and the list will be updated on each invocation.

Standalone Installation

Fix Rook-Ceph OSD not getting assigned if underlying node changes

Third-party dependencies

kdb Insights Enterprise 1.5.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.5.0 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.5.0.tgz
Operator	kxi-operator-1.5.0.tgz
CLI	kxicli-1.5.0-py3-none-any.whl
C SDK	kxi-c-sdk 1.5.0
Java SDK	kxi-java-sdk 1.5.0
RT Bridge	rt-bridge 1.5.0
Infrastructure	kxi-terraform-1.5.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly from 1.3.x to any newer release, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

Stop all external publishers to kdb Insights Enterprise
Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

Backup any assemblies that may be running

$ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

Teardown assemblies named in the output of the backup command

$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

Delete the RT PVCs

kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

Carry out the upgrade

Restore assemblies

kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

Restriction on upgrading to older versions

The CLI will not allow you to "upgrade" to an older version of kdb Insights Enterprise. Please use the rollback command to revert to a previous version.

Known Issues

LicensingPublishingStorage ManagerCLIResource CoordinatorQueriesPipelinesRollbackReinstall

On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.
A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below
```
klic license renew $LICENSE_UUID --version=latest

env QLIC=$(pwd) klic license fetch $LICENSE_UUID
```
Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.
```
printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml

kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml
```

Publishing of a single message larger than 1GB is not supported.

Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. Therefore, there is a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:
- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit super back to sp
- Press the down arrow to go to the next line

Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

A known issue exists within the kxi install rollback command when rolling back to prior versions. When the target version was installed with keycloak.initUser enabled. It is expected the keycloak-config-cli job will fail. This is due to the rollback trying to revert that user's password and violating the password history policy.

The rollback failure will manifest as the failure of the keycloak-config-cli job. This will create job pods as it fails and retries. In order to proceed with the rollback it's recommended to delete kubectl delete job ${RELEASE_NAME}-keycloak-config-cli.

A known issue exists during a reinstall using the CLI where the keycloak.initUser is enabled. If the user's password has changed, the reinstall will try to revert it back and be rejected by the password history policy.

This can be avoided by disabling the user import on reinstall using --import-users False as below:

kxi install run --import-users False ..

Backward Compatibility

If you are upgrading from a version earlier than 1.5.0 you must also read each of the relevant release notes below to ensure you know all the changes that will affect the upgrade.

1.4.5

Release for kdb Insights Enterprise.

Release Date

2023-04-26

Improvements

[IMPROVEMENT]: If publishing a message larger than 1GB, the publisher will now receive a message too large error. Users can catch the error (need more clear detail) to avoid getting an error at runtime. This would previously cause downstream subscriber processes to run out of memory during processing.

[IMPROVEMENT]: SQL queries that filter by time but do not explicitly select the time column are now supported. This would previously error.

Fixes

[FIX]: Fixed an issue where rapid successive queries using qsql such as those from Views in the kdb Insights UI could return query results out of order.

[FIX]: Schema conversions are now more robust on network based file systems (ex. Rook Ceph). Previously, this could result in a EXDEV. OS Reports: Invalid cross-device link error in the Storage Manager.

[FIX]: Window nodes in the Stream Processor can now accept dictionaries as input batches. Previously, incoming dictionaries would be merged with input batch in the buffer, instead of appending to a buffered list of dictionaries.

Third-party dependencies

kdb Insights Enterprise 1.4 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.4.5.tgz
Operator	kxi-operator-1.4.2.tgz
CLI	kxicli-1.4.2-py3-none-any.whl
C SDK	kxi-c-sdk 1.4.0
Java SDK	kxi-java-sdk 1.4.0
RT Bridge	rt-bridge 1.4.0
Infrastructure	kxi-terraform-1.4.1.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly created using a version prior to 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

Stop all external publishers to kdb Insights Enterprise
Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

Backup any assemblies that may be running

$ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

Teardown assemblies named in the output of the backup command

$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

Delete the RT PVCs

kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

Carry out the upgrade

Restore assemblies

kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

Licensing

A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below.

klic license renew $LICENSE_UUID --version=latest
env QLIC=$(pwd) klic license fetch $LICENSE_UUID

Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.

printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml
kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Monitoring

We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes.

If you wish to continue with the higher frequency you can take one of the following steps:

-  Update the interval field within their charts.

    ```yaml
    metrics:
      serviceMonitor:
        interval: "30s"
    ```

- Override the chart values using the global value within insights/values.yaml.

    ```yaml
    global:
      metrics:
        serviceMonitor:
          interval: "30s"
    ```

- Update your assembly yaml, to override the default.

    ```yaml
    spec:
      metrics:
        enabled: true
        serviceMonitor:
          enabled: true
          interval: 10s
    ```

- Update your default within the Assembly Resource - This is generated using the insights/values.yaml.

    ```yaml
    kxi-operator:
      config:
        sidecar:
          metrics:
            serviceMonitor:
              interval: 30s
    ```

Default password policy

The default password policy has been updated to include the following rule:

Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

PublishingLicensingStorage ManagerCLIResource CoordinatorQueriesImport WizardKXI ControllerPipelinesViewsAccessibilityPostgreSQL

Publishing of a single message larger than 1GB is not supported.

On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:
- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit super back to sp
- Press the down arrow to go to the next line

If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.

"[] Failed to compare saved assembly  to deployed version, error: type"

Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

Screen readers will not read out node information represented in a pop-up when hovered over, and may freeze when attempting to read a large amount of content.
Unable to navigate up and down cells and read all table content when navigating using only a keyboard.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml.

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.4

Release for kdb Insights Enterprise.

Release Date

2023-04-20

Improvements

[IMPROVEMENT]: The kdb Insights Database now supports a REST interface for loading historical datasets into an existing database. You can now perform a historical batch ingest using a simple REST API. For more details, see batch ingestion.

[IMPROVEMENT]: Query requests that supply a log correlator (logCorr) will now see that correlator in the Service Gateway logs, allowing for easier traceability in query execution and an improved debugging experience. Previously a random correlator was used in the Service Gateway logs.

[IMPROVEMENT]: Added significant improvements for accessibility using keyboard navigation and/or screen readers in the UI.

Users can exit a text area in a code block on the Query page by first pressing "CTRL+M" before pressing the "Tab" key.
The entity tree menu button can now be opened when hovering your mouse over the entity tree. Arrows can then be used to navigate.
"Code View" on the Schemas page can now be closed with the "Esc" key.
The Views and Pipeline pages now allow draggable elements to be inserted/dragged onto the page with the use of your keyboard.

Fixes

[FIX]: Security vulnerabilities patched in the Stream Processor for the following CVEs: CVE-2022-41910, CVE-2022-41902, CVE-2022-41900, CVE-2022-41880.

[FIX]: Additional historical tiers that use object storage in the UI now allow dashes and spaces in file names. Previously they would error for dashes and spaces.

[FIX]: Migrating tiers to object storage could fail with the eventual error SMDBM Migration discontinued due to failure if multiple threads were used.

[FIX]: If an SM client cannot connect to SM after a number of retries, it will now restart the pod to try to automatically recover. This resolves connection issues in DAPs when there are network connectivity issues.

[FIX]: Reading of files from Azure Storage larger than ~1MB now successfully writes to a kdb Insights database. Previously this was erroring due to an issue in the chunking of larger files.

[FIX]: The Service Gateway was creating duplicate connections to Resource Coordinators while they were waiting to come online. This was resulting in a large number of redundant connections being opened between the services. In large deployments, this could result in connection errors in the Resource Coordinator.

[FIX]: Expression readers now return the result of Python lambdas.

[FIX]: After upgrading kdb Insights Enterprise from 1.3.x to 1.4.x, clicking a node in a previously-saved pipeline in the UI no longer causes the function definition to flicker and disappear.

[FIX]: Fixed issue with 1.4.0 kxi-terraform scripts, where Terragrunt config was incompatible and was causing deployment failures on AWS.

Third-party dependencies

kdb Insights Enterprise 1.4 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4 release of the standalone infrastructure installation scripts:target="_blank"}.

Artifacts

type	location
Enterprise	insights-1.4.4.tgz
Operator	kxi-operator-1.4.2.tgz
CLI	kxicli-1.4.2-py3-none-any.whl
C SDK	kxi-c-sdk 1.4.0
Java SDK	kxi-java-sdk 1.4.0
RT Bridge	rt-bridge 1.4.0
Infrastructure	kxi-terraform-1.4.1.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly created using a version prior to 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

Stop all external publishers to kdb Insights Enterprise
Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

Backup any assemblies that may be running

$ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

Teardown assemblies named in the output of the backup command

$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

Delete the RT PVCs

kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

Carry out the upgrade

Restore assemblies

kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

Licensing

A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below.

klic license renew $LICENSE_UUID --version=latest
env QLIC=$(pwd) klic license fetch $LICENSE_UUID

Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.

printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml
kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Monitoring

We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes.

If you wish to continue with the higher frequency you can take one of the following steps:

-  Update the interval field within their charts.

    ```yaml
    metrics:
      serviceMonitor:
        interval: "30s"
    ```

- Override the chart values using the global value within insights/values.yaml.

    ```yaml
    global:
      metrics:
        serviceMonitor:
          interval: "30s"
    ```

- Update your assembly yaml, to override the default.

    ```yaml
    spec:
      metrics:
        enabled: true
        serviceMonitor:
          enabled: true
          interval: 10s
    ```

- Update your default within the Assembly Resource - This is generated using the insights/values.yaml.

    ```yaml
    kxi-operator:
      config:
        sidecar:
          metrics:
            serviceMonitor:
              interval: 30s
    ```

Default password policy

The default password policy has been updated to include the following rule:

Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

LicensingStorage ManagerCLIResource CoordinatorQueriesImport WizardKXI ControllerPipelinesViewsAccessibilityPostgreSQL

On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
When writing python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:
- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit super back to sp
- Press the down arrow to go to the next line

If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.

"[] Failed to compare saved assembly  to deployed version, error: type"

Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

Screen readers will not read out node information represented in a pop-up when hovered over, and may freeze when attempting to read a large amount of content.
Unable to navigate up and down cells and read all table content when navigating using only a keyboard.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml.

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.3

Release Date

2023-03-31

Improvements

[IMPROVEMENT] We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes.

If you wish to continue with the higher frequency you can take one of the following steps:

-  Update the interval field within their charts.

    ```yaml
    metrics:
    serviceMonitor:
        interval: "30s"
    Assemblies
    ```

Update your assembly yaml, to override the default.

```yaml
    spec:
        metrics:
            enabled: true
            serviceMonitor:
            enabled: true
            interval: 10s
```

Update your default within the Assembly Resource - This is generated using the insights/values.yaml.

```yaml
kxi-operator:
config:
    sidecar:
    metrics:
        serviceMonitor:
        interval: 30s
```

Fixes

[FIX]: Service Gateway: fixed remote code injection vulnerability in Snake YAML CVE-2022-1471

[FIX]: Queries to the kdb Insights Database now have a default timeout of 60s (previously 30s). This allows for longer running queries to be issued from query clients, such as the scratchpad.

[FIX]: Additional historical tiers in the UI now allow for a retention period to be set on the HDB tier. Previously this was erroring due to a validation issue.

[FIX]: In the UI, the real-time database rollover tier now correctly checks that rollover is less than the interval tier.

[FIX]: Requests to object storage (Azure Storage, AWS S3, Google Cloud Storage) will now backoff and retry if they get a throttle request from the vendor. Jitter will be introduced to the retry to ensure that replicated processes do not all retry at the same time.

[FIX]: Setting a column type to Any now correctly uses an anymap type. This was previously reset to an unselected value.

[FIX]: The REST Server's heartbeat logs are now set to TRACE to reduce log output (previously they were logged as INFO logs).

[FIX]: Pasting into a pipeline node’s configuration no longer duplicates the node.

[FIX]: Resolved issue where some pipelines with merge nodes would error with 'missing edge'.

[FIX]: Comments can now be added to the last line of the Scratchpad without causing it to error.

[FIX]: The Scratchpad now has more memory allocated for ad-hoc queries in the kdb Insights Enterprise query panel.

[FIX]: Stream Processor file reader nodes (Azure, AWS S3, File and Google storage) would only process the first file presented even when multiple files were present.

[FIX]: Object storage tiers can now be added to a database via the kdb Insights Enterprise UI.

Third-party dependencies

kdb Insights Enterprise 1.4.3 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.3 release of the standalone infrastructure installation scripts.

Artifacts

type	location

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

Stop all external publishers to kdb Insights Enterprise
Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

Backup any assemblies that may be running

$ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

Teardown assemblies named in the output of the backup command

$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

Delete the RT PVCs

kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

Carry out the upgrade

Restore assemblies

kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule:

Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

LicensingStorage ManagerCLIResource CoordinatorQueriesImport WizardKXI ControllerPipelinesViewsPostgreSQL

On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.
Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
When writing python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:
- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit super back to sp
- Press the down arrow to go to the next line

If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.

"[] Failed to compare saved assembly  to deployed version, error: type"

Please ensure all pipelines have different names. Two pipelines with the same name may not be deploy correctly.
Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml.

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.2

Release for kdb Insights Enterprise.

Release Date

2023-03-17

Fixes

[FIX] Resource utilization in the UI Scratchpad has been greatly improved. This fix along with a larger default memory allocation provide greater Scratchpad process stability.

Third-party dependencies

kdb Insights Enterprise 1.4.2 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.4.2.tgz
Operator	kxi-operator-1.4.0.tgz
CLI	kxicli-1.4.1-py3-none-any.whl
C SDK	kxi-c-sdk 1.4.0
Java SDK	kxi-java-sdk 1.4.0
RT Bridge	rt-bridge 1.4.0
Infrastructure	kxi-terraform-1.4.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

Stop all external publishers to kdb Insights Enterprise
Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

Backup any assemblies that may be running

$ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

Teardown assemblies named in the output of the backup command

$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

Delete the RT PVCs

kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

Carry out the upgrade

Restore assemblies

kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule:

Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

LicensingStorage ManagerCLIResource CoordinatorQueriesImport WizardKXI ControllerPipelinesViewsPostgreSQL

On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.
Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
When writing python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:
- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit super back to sp
- Press the down arrow to go to the next line

If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.

"[] Failed to compare saved assembly  to deployed version, error: type"

When creating a pipeline the user interface does not stop you from creating it with the same name as an existing pipeline, however two pipelines with the same name may not be deploy correctly. Please ensure all pipelines have different names.
Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml.

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.1

Release for kdb Insights Enterprise.

Release Date

2023-03-14

Fixes

[FIX] In 1.3.3 and 1.4.0, if the Storage Manager falls behind on EODs for more than a full day, IDB links could be written incorrectly when late data was present. If this issue is encountered, manual data changes are required to recover the database including backup of the stream logs from the date of the failure. Please contact KX Support for help with recovery.

This issue is characterized by a log statement from the Storage Manager as follows: "Unexpected condition: EOXa watermark date YYYY.MM.DD is less than HDB stream date YYYY.MM.DD, setting it to match HDB stream date". The IDB may also start reporting "No such file or directory" on ordinal partitions.

Third-party dependencies

kdb Insights Enterprise 1.4.1 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts.

Artifacts

type	location
Enterprise	insights-1.4.1.tgz
Operator	kxi-operator-1.4.0.tgz
CLI	kxicli-1.4.1-py3-none-any.whl
C SDK	kxi-c-sdk 1.4.0
Java SDK	kxi-java-sdk 1.4.0
RT Bridge	rt-bridge 1.4.0
Infrastructure	kxi-terraform-1.4.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

Stop all external publishers to kdb Insights Enterprise
Wait for all published messages to be available for querying from the database - this will reduce (but not eliminate) the amount of duplicate data

Backup any assemblies that may be running

$ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

Teardown assemblies named in the output of the backup command

$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

Delete the RT PVCs

kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

Carry out the upgrade

Restore assemblies

kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule:

Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

LicensingStorage ManagerCLIResource CoordinatorQueriesImport WizardKXI ControllerPipelinesViewsPostgreSQL

On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

If end-of-day processing is interrupted and remains incomplete until the next EOD period, this prevents further EODs from completing and requires manual recovery.
Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.
Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:
- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit super back to sp
- Press the down arrow to go to the next line

If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.

"[] Failed to compare saved assembly  to deployed version, error: type"

Please ensure all pipelines have different names. Two pipelines with the same name may not be deploy correctly.
Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml.

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.0

Release for kdb Insights Enterprise.

Release Date

2023-03-03

New Features

Introducing Pythonic pipeline development in the UI, dynamic logging notifications providing direct access to root cause of underlying issues, an enhanced user experience for database configuration and additional security enhancements.

[NEW] Pipelines

You can now use Python in the pipeline builder in the UI to perform aggregations and analytics.
Ingest large batches of data directly into the historical database has been moved out of beta.
- Use fewer resources. Memory and disk resources greatly reduced during ingest.
- Select the Write Direct to HDB field in the Database Writer in the UI, or use the q and Python APIs.

[NEW] User Interface

Improvements to the observability user experience with two new icons in top right hand-side of the UI banner:
- A notification icon shows a count of new log messages. This ensures you are notified of any new error messages wherever you are in the UI. You can also click on the icon to show more details of the message, these are displayed in the right-hand third of the screen.
- A settings icon enables you to update the application logging level for future log records written by the kdb Insights Enterprise components.
You can now share the URL of a View by selecting the view on the left-hand toolbar, right-clicking and choosing the new Share button. This URL just shares the content of the View and not the toolbars (e.g. it does not show the left-hand toolbar or the Design/Preview panel at the top).

[NEW] Databases

A new version of our Database Configuration page is now available, under a beta flag, that provides an enhanced user experience. It can be toggled on and off from the Settings option in the main toolbar. We recommend that you use the existing Database Configuration age for all running databases, but please try the new user interface out and give us your feedback.

[NEW] Packages

The user interface now allows you to configure which Packages and their associated Custom APIs are used within individual databases, this had previously only been possible through kdb CLI deployments.

[NEW] Security Improvements

Keycloak has been updated to version 19. This impacts the following:
- The Keycloak Admin console has been changed
- Logout screen provides a redirection to the login page

Improvements

Keycloak realm

In this release, the handling of the Keycloak realm has been improved. This realm contains the role definitions, users and clients. Previously the realm was not being imported by default on install. The role definitions will always be imported as part of an install or upgrade now. The users and clients will only be imported by default on install (kxi install), and not on upgrade.

To make changes to the initClient and initUser on upgrade, you will need to tell the application to import them using the setting below.

keycloak:
  importUsers: true

Reliable Transport

The kdb Insights Enterprise transport layer has adopted a version of Raft called NuRaft. This improves the fault tolerance and performance of kdb Insights Enterprise. NuRaft is an open source version of Raft. Migrating to NuRaft also provides kdb Insights Enterprise with additional capabilities for future enhancements to Reliable Transport.

Third-party dependencies

kdb Insights Enterprise 1.4.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts .

Artifacts

type	location
Enterprise	insights-1.4.0.tgz
Operator	kxi-operator-1.4.0.tgz
CLI	kxicli-1.4.0-py3-none-any.whl
C SDK	kxi-c-sdk 1.4.0
Java SDK	kxi-java-sdk 1.4.0
RT Bridge	rt-bridge 1.4.0
Infrastructure	kxi-terraform-1.4.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

Stop all external publishers to kdb Insights Enterprise
Wait for all published messages to be available for querying from the database - this will reduce (but not eliminate) the amount of duplicate data

Backup any assemblies that may be running

$ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

Teardown assemblies named in the output of the backup command

$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

Delete the RT PVCs

kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

Carry out the upgrade

Restore assemblies

kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule

Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps .

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Supported Python versions in the CLI

Support for Python 3.7 has been dropped in the CLI.

Known Issues

LicensingStorage ManagerCLIResource CoordinatorQueriesImport WizardKXI ControllerPipelinesViewsPostgreSQL

On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

If end-of-day processing is interrupted and remains incomplete until the next EOD period, this prevents further EODs from completing and requires manual recovery.
Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.
Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:
- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit super back to sp
- Press the down arrow to go to the next line

If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.

"[] Failed to compare saved assembly  to deployed version, error: type"

When creating a pipeline the user interface does not stop you from creating it with the same name as an existing pipeline, however two pipelines with the same name may not be deploy correctly. Please ensure all pipelines have different names.
Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml.

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.3.4

Release for kdb Insights Enterprise.

Release Date

2023-03-13

Fixes

[FIX] As of 1.3.3, if the Storage Manager falls behind on EODs for more than a full day, IDB links could be written incorrectly when late data was present. If this issue is encountered, manual data changes are required to recover the database including backup of the stream logs from the date of the failure. Please contact KX Support for help with recovery.

This issue is characterized by a log statement from the Storage Manager as follows: "Unexpected condition: EOXa watermark date YYYY.MM.DD is less than HDB stream date YYYY.MM.DD, setting it to match HDB stream date". The IDB may also start reporting "No such file or directory" on ordinal partitions.

Artifacts

type	location
Infrastructure	kxi-terraform-1.3.1.tgz
kdb Insights Enterprise	insights-1.3.4.tgz
Operator	kxi-operator-1.3.1.tgz
CLI	kxicli-1.3.1-py3-none-any.whl
ODBC Driver	kodbc 1.3.0
Java SDK	java-sdk 1.3.0

Batch Ingest Known Issues

In the unlikely event of two batch ingests being done within an end of interval (EOI) event (for example, a 10 min window) a one hour delay would occur for the subsequent ingest, but there would not be any data loss, just a delay.
If the Storage Manager (SM) crashes during a batch ingest, the ingest will fail. On SM recovery, it will not try again and must be restarted, but no clean-up is required.

Upgrade Notes

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Resource Coordinator

If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" can be misleading in the timeout message. It should be interpreted as "Timed out before we were able to dequeue".
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

Python Scratchpad for Queries

If the script consists of multiple statements, the console does not print the output of the last evaluation, and instead it only displays the expression before it is evaluated.

S3 Import Wizard

If you select the source, S3, and click Next, no error will be thrown, which is invalid. At least one file needs to be present.

Keycloak Config CLI

We recommend that you enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default Password Policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

At least one uppercase letter
At least one lowercase letter
At least one symbol
At least one number
Minimum length of 14 characters

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the Insights values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

If you currently set initUser.enabled=true or initClient.enabled=true within your own values.yaml, you may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where a Keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, you must set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly `blockSize` Changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
- this has the highest performance, but has no RAM limit on received data
if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
- the smaller this number is set to, the worse ingest performance but stronger RAM limits; this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

LicensingStorage ManagerDiscovery ServiceAPI GatewayUI Logout

On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.

sm:
  size: 1

If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.

Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:

helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>

Note that despite the command name, this operation does not rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

After performing a logout action, you will not be redirected back to the login screen.
If you were logged out due to inactivity, you might see a "Logout failed" error.

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}.

1.3.3

Release for kdb Insights Enterprise.

Release date

2023-01-17

New features

Functionality for improved ingestion, efficiency and reduction of resources. Take advantage of an IDE to develop your databases, pipelines and query interfaces.

[NEW] Improved ingestion

[Beta] Ingest large batches of data directly into the historical database:
- Use fewer resources. Memory and disk resources greatly reduced during ingest.
- Select the Direct Writing field in the Database Writer in the UI, or use the q and Python APIs.
See details here for known issues.

[NEW] Writing data to Amazon S3

You can now write data to Amazon S3 by either:
- Selecting the Amazon S3 Writer node in the UI.
- Using the q and Python APIs.

[NEW] Improved IDE capabilities

Build YAML files and upload them using kdb Insights CLI:
- Speed up development workflow, improve productivity.
- Access to the Kubernetes control plane is no longer needed. This is now handled for you by the CLI.
- Assembly management using CLI.
If you have access to KX Analyst, you can use it to develop functions. You can now:
- Enjoy a rich IDE development environment with syntax highlighting for YAML files.
- Query kdb Insights Enterprise data using REST API or query interface.
- Upload using the CLI or as part of CI/CD pipeline.
- Build a complete SDLC that harnesses the flexibility of KX Analyst against kdb Insights Enterprise data.
Note

Contact your KX Sales representative if you are interested in using KX Analyst. In future releases, additional IDEs will become available, including VS Code and Jupyter Notebooks.

Improvements

[NEW] Reliable Transport

Reliable Transport's logging default level for the sequencer and merger has been changed from DEBUG to INFO.

Fixes

[FIX] Following a hard reset of the Reliable Transport (RT), the log files in the out folder are truncated rather than deleted. This means that the replicated log files for all subscribers to the RT instance will be garbage collected once the RT archiver starts after the reset. This prevents a hard reset from causing subscriber PVCs to fill up due to old/corrupt log files remaining.

Artifacts

type	location
Infrastructure	kxi-terraform-1.3.1.tgz
kdb Insights Enterprise	insights-1.3.3.tgz
Operator	kxi-operator-1.3.1.tgz
CLI	kxicli-1.3.1-py3-none-any.whl
ODBC Driver	kodbc 1.3.0
Java SDK	java-sdk 1.3.0
RT Bridge	rt-bridge-1.4.0-install.sh

Known Issues

Batch ingest

In the unlikely event of two batch ingests being done within an end of interval (EOI) event (for example, a 10 min window) a one hour delay would occur for the subsequent ingest, but there would not be any data loss, just a delay.
If the Storage Manager (SM) crashes during a batch ingest, the ingest will fail. On SM recovery, it will not try again and must be restarted, but no clean-up is required.

Discovery Service

If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.

Upgrade Notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Terraform Script

Note the version. GKE cluster cannot be deployed using a 1.3.0 version of the script; a 1.3.* version of the client scripts is needed to use Kubernetes v1.22 on GKE.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

At least one uppercase letter
At least one lowercase letter
At least one symbol
At least one number
Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the Keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

If you currently set initUser.enabled=true or initClient.enabled=true within your values.yaml, you may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where Keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly `blockSize` changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

If unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval. This has the highest performance, but has no RAM limit on received data
If set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM. The smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

LicensingStorage ManagerAPI GatewayUI LogoutResource CoordinatorPython Scratchpad for QueriesS3 Import Wizard

On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.

sm:
  size: 1

Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:

helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>

Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

After performing a logout action, the user will not be redirected back to the login screen.
In the case a user was logged out due to inactivity, a "Logout failed" error might be observed.

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

If the script consists of multiple statements, the console does not print the output of the last evaluation, and instead it only displays the expression before it is evaluated.

If S3 is selected as the source and Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

1.3.2

Release for kdb Insights Enterprise.

Release Date

2022-12-05

New Features

[NEW] Pipelines

[Beta] Compressed files (GZIP) can now be decoded using the q and Python APIs.
Multiple file paths can be added to a cloud storage Reader node using the + button in the UI, or via the q and Python APIs.
Wildcard support using * is now available when using a single file path with the Amazon S3 Reader.
CSV encoding is now available, using a new node in the UI, or via the q and Python APIs.
It is now possible to write to Amazon S3 buckets, using a new node in the UI, or via the q and Python APIs.
Pipeline lifecycle events can be subscribed to via the q API. This can be used, for example, to invoke a callback once files have been processed by a pipeline.

Improvements

Queries will no longer time out after 60 seconds at the gateway. Longer timeouts can be configured in getData API calls using the timeout header option.

Fixes

[FIX] Query

An issue where DAPs could enter low-memory mode earlier than they should has been resolved.

Artifacts

type	location
Infrastructure	kxi-terraform-1.3.0.tgz
Platform	insights-1.3.2.tgz
Operator	kxi-operator-1.3.1.tgz
CLI	kxicli-1.3.0-py3-none-any.whl
ODBC Driver	kodbc 1.3.0
Java SDK	java-sdk 1.3.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

At least one uppercase letter
At least one lowercase letter
At least one symbol
At least one number
Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within kdb Insights Enterprise values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

Users who currently set initUser.enabled=true or initClient.enabled=true within their own values.yaml may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, the user is required to set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly `blockSize` changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
- this has the highest performance, but has no RAM limit on received data
if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
- the smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

LicensingStorage ManagerDiscovery ServiceAPI GatewayUI Logout

On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.

sm:
  size: 1

If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.

Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:

helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>

Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

After performing a logout action, the user will not be redirected back to the login screen.
In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.3.1

Release for kdb Insights Enterprise.

Release Date

2022-11-23

Security

[FIX] kdb Insights Enterprise 1.3.1 contains security updates for a number of critical CVEs

Streams and Reliable Transport

[FIX] An issue which could cause RT archiving to fail has been fixed.

Artifacts

type	location
Infrastructure	kxi-terraform-1.3.0.tgz
Platform	insights-1.3.1.tgz
Operator	kxi-operator-1.3.1.tgz
CLI	kxicli-1.3.0-py3-none-any.whl
ODBC Driver	kodbc 1.3.0
Java SDK	java-sdk 1.3.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

At least one uppercase letter
At least one lowercase letter
At least one symbol
At least one number
Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights Enterprise values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

Users who currently set initUser.enabled=true or initClient.enabled=true within their own values.yaml may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, the user is required to set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly `blockSize` changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
- this has the highest performance, but has no RAM limit on received data
if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
- the smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

LicensingStorage ManagerDiscovery ServiceAPI GatewayUI Logout

On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.

sm:
  size: 1

If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.

Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:

helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>

Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

After performing a logout action, the user will not be redirected back to the login screen.
In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.3.0

Release for kdb Insights Enterprise.

Release Date

2022-10-26

Features

[NEW] User Interface

Items in the page tree have been renamed for clarity and consistency with the CLI.
- Data is now known as Assemblies
- Explore is now known as Queries
- Reports is now known as Views

[NEW] Pipelines

An "Auto" setting for Parse Strings in the Apply Schema node has been added. The column will be parsed if the input is textual (strings or bytes), but the output type is non-textual.
Data types are now visible for each column for Test Deploy previews. These can be used to validate the data types and the data shape of Transforms and Maps results.
Data types and column names are validated for Database writers as part of a test deploy.
Additional Reader validation has been introduced to pipelines, to ensure correctness of settings before deploying.
Test deploy previews can now show non-tabular data.
Previews are now available for Reader nodes, before data has been decoded.
The visibility of a Pipeline's state has been improved in the Overview tab. Warning and error counts can be clicked on to launch a diagnostics tab for that Pipeline.

[NEW] Assemblies

It is now possible to see assemblies created using kxi assembly create in the UI. Note that assemblies created and managed through the command-line are not editable via the UI.
Additional logging has been added to the Storage Manager, to ease troubleshooting.
Schema configuration validation has been improved.

[NEW] Install and Packages

The kdb Insights Operator is no longer uninstalled by default by kxi install delete, the command line parameter --uninstall-operator must now be used.
Many values for kxi install setup can now be passed via command line parameter, please see the CLI Reference for more detail.
Improved support for rolling back to previous versions. Enter a previous version number using the upgrade command kxi install upgrade to roll back.
The current kubectl context is now used as the default in all commands, instead of the namespace specified in the cli-config file.
kdb Insights Python modules for packaging and machine learning are no longer dependent on libc.musl, and can now be used for development on Windows and Mac.
Custom user query APIs and Aggregation functions can be installed and initialised within Data Access Processes and Aggregators as outlined here.
Custom user query APIs and Aggregation functions can be installed and initialised within Data Access Processes and Aggregators as outlined here.

[NEW] Observability

Reliable Transport diagnostics are now available in diagnostics exports from the UI.

[NEW] Infrastructure

The base image for most components has been moved to Rocky Linux.

Fixes

A large number of fixes have gone into kdb Insights Enterprise 1.3.0. Known issues which have been resolved, along with some other highlights, are listed below.

[FIX] Streams and Reliable Transport

A hard reset can now be performed on Reliable Transport, to recover from stream corruption. Please contact KX Support for usage instructions.

[FIX] Pipelines

Pipelines which have their data cleared when they're torn-down can now be re-deployed with the same name. Previously, this would cause RT sequence number problems, stopping data from flowing.
Persistent pop-up errors in the UI after resizing a Map node have been resolved.
.qsp.read.fromDatabase can now be used in pipelines.
Renaming a pipeline node no longer disconnects its edges.
Empty pipelines are no longer deployable.

[FIX] User Interface

Validation fixes so that date and time fields can be entered without use of the picker.

[FIX] Java SDK

Occasional authentication failures when querying via Java SDK have been resolved. These would previously be seen more frequently on slower connections.

Artifacts

type	location
Infrastructure	kxi-terraform-1.3.0.tgz
Platform	insights-1.3.0.tgz
Operator	kxi-operator-1.3.0.tgz
CLI	kxicli-1.3.0-py3-none-any.whl
ODBC Driver	kodbc 1.3.0
Java SDK	java-sdk 1.3.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of KX Insights.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

At least one uppercase letter
At least one lowercase letter
At least one symbol
At least one number
Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights Enterprise values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

Users who currently set initUser.enabled=true or initClient.enabled=true within their own values.yaml may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, the user is required to set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Known Issues

Service GatewayUser InterfaceLicensingStorage ManagerDiscovery ServiceAPI GatewayUI Logout

If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below

   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

The user-defined variables & functions in the Queries tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)
The Queries tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*
The Queries tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.
On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.

sm:
  size: 1

If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here

Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:

helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>

Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

After performing a logout action, the user will not be redirected back to the login screen.
In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.2.1

Release for kdb Insights Enterprise.

Release Date

2022-09-22

Fixes

kdb Insights Enterprise 1.2.1 is a security update for vulnerabilities in the included images for Keycloak/PostgreSQL.

Artifacts

type	location
Infrastructure	kxi-terraform-1.2.0.tgz
Platform	insights-1.2.1.tgz
Operator	kxi-operator-1.2.0.tgz
CLI	kxicli-1.2.0-py3-none-any.whl
ODBC Driver	kodbc 1.2.0
Java SDK	java-sdk 1.2.0
keycloak-server	keycloak-server-0.2.1.tgz

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Version 1.2.1 requires a custom storageClassName definition

The addition of support for packages introduces a new storageClassName defined as sharedfiles, this is defined by default in any deployments of kdb Insights Enterprise that rely on the KX provided Terraform scripts for deployment but must be added manually by a user as outlined here.

Alternatively, within the deployment configuration as outlined here, modify the storageClassName to an alternative supported RWX storage option for example rookcephfs.

Keycloak 18

The version of Keycloak used within kdb Insights Enterprise 1.2.0 and 1.2.1 has been upgraded from Keycloak 16 to Keycloak 18.

Please follow the Keycloak 18 migration steps to migrate your Keycloak data prior to upgrading kdb Insights Enterprise from versions older than 1.2.0.

Azure Marketplace upgrade

Upgrading from v1.2.0 Marketplace release has a known issue. Keycloak was upgraded in v1.2.1 which contained non backward compatible changes. The PostgreSQL secret format has changed, and existingSecret from values.yaml has an extra nested key. Please make sure values.yaml has the extra auth nested key, and that the secret containing PostgreSQL passwords, usually called kxi-postgresql, has dropped the postgresql prefix from the data keys.

Known Issues

Service GatewayUser InterfaceLicensingSDKsStorage ManagerStream ProcessorAPI GatewayUI Logout

If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below

   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)
The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*
The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.
On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.
When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data

Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.

sm:
  size: 1

Torn-down pipelines that include Database or Stream Writers cannot be directly restarted after having their resources removed. Upon redeploy, data will not be published downstream from the pipeline. A workaround is to rename the pipeline. Resources are removed by checking "Clear Pipeline State" during teardown in the UI, or by directly deleting the Kubernetes PVCs backing the pipeline. If either of these things are done, the pipeline must be renamed before relaunching it.
.qsp.read.fromDatabase cannot be run in a pipeline to query a kdb Insights Enterprise database. The following example code can be used as a workaround to query data (ex. reference data to be used within a pipeline).

onResponse:{[hdr;data]
if[0 <> hdr`ac;
    .sp.log.error ("Query failed host=%s, api=%s, code=%n, error=%s";hdr`gw; hdr`api; hdr`ac; hdr`ai)];
publish data
}
.qsp.onStart {
    h:hopen hsym `$.spenv.getReleaseName[],"-sg-gateway:5050";
    res: neg[h] (`.kxi.getData; (enlist `table)!(enlist `instruments); `onResponse; ()!());
}
.qsp.run .qsp.read.fromCallback[`publish] .qsp.write.toConsole[]

Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:

helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>

Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

After performing a logout action, the user will not be redirected back to the login screen.
In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can navigate re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.2.0

Release for kdb Insights Enterprise.

Release Date

2022-09-02

Features

[NEW] Install and Packages

Upgrade support{:target="blank"} in the kxi command line tool for easier upgrades. _Please take note of other Upgrade Notes below.
You must now provide a namespace when deleting installs using kxi install delete in the CLI.
Ability to list, add and delete custom versioned user code via packages to kdb Insights Enterprise via the CLI outlined here.
Support within packages for the definition of User Defined Functions (UDFs) written in Python or q for deployment to stream-processor or use in the Explore Window.

[NEW] Pipelines

MQTT QoS 0 Reader has been added.
Users can run Test deploys in the UI, which provide user feedback on issues when building or editing a pipeline. This includes visualization of the data at each node in the pipeline, and clear indications of which nodes failed.
CPU and Memory resources can now be configured from the User Interface to allow rightsizing of data pipelines.
Use of User Defined Functions as part of a pipeline.
Apply Schema is now available for use after the CSV decoder, so the optional schema parameters for the CSV node have been removed from the Pipelines UI. Users should take care to ensure they're parsing all fields in the Apply Schema node when it's being applied to CSV decoded data.

[NEW] Explore and Query

Significant speed improvements to Explore. Query environments are pre-allocated as part of Data/assemblies, and so switching between tabs, assemblies, or instances is now instant. Query environments must be enabled in the Database/Data Access screen, or in assembly configuration, for the assembly to be available in Explore.
Filtering and aggregation are now available on the Query tab.
There is no need to choose a database when using the Query or SQL options in the Explore window. Label filtering can be used to select particular databases. You can now read from tables with the same name and schema across multiple databases when using the Query or SQL tabs.
Import and use of User Defined Functions in pipelines and the Explore Window to allow running of custom code.
Console output can now be cleared by right-clicking in the Console and choosing "Clear".
The query section at the top of Explore tabs can now be collapsed for a larger code area.

[NEW] Databases

CPU and Memory resources can now be configured from the User Interface to allow rightsizing of storage and query services.
Redundant replicas possible for all query-path processes, enabling greater query uptime despite process, node, or availability-zone failure.
Ability for Data Access tiers (RDB, IDB, HDB) to share compute resources by using mountList within Assembly configuration rather than mountName.
Granular assembly deployment status showing the state of deploying components, and assembly deployment errors, are now surfaced in the UI. Mouse-over the assembly's status icon while it's deploying to see the breakdown.
Environmental variables can now be configured in the UI for Data Access and Storage when the Advanced view is enabled.
Code View for Schemas is now available in the UI. Users can edit (or copy/paste) schemas directly, rather than configuring through UI forms.

[NEW] Performance

Cluster overprovisioning support that can be configured to enable faster scaling, by reserving resources.

[NEW] Java SDK

Query support has been added to the Java SDK.
For ingestion there is an option to validate the data being ingested to ensure it is in the format that is expected by the database, before it is ingested.
Batch publishing is available, if required.

[NEW] Observability

It is now possible to filter for a displayed value (for example, a particular Container) in the Diagnostics screen. Right-click on the value and select "Filter by [...]".

[NEW] Infrastructure

A storage class named sharedfiles, that is backed by shared storage, is automatically created when deploying via client Terraform scripts. sharedfiles storage is used for storing Packages and required by kdb Insights Enterprise.

Fixes

A large number of fixes have gone into kdb Insights Enterprise 1.2.0. Known issues which have been resolved, along with some other highlights, are listed below.

[FIX] Pipelines

Pipelines no longer report an initial "Unresponsive" status in the UI when launching.

[FIX] Explore and Query

With the introduction of Query Environments, there is no longer an issue where PVCs are not removed upon assembly teardown due to active Explore tabs targeting the assembly.
SQL queries can now target splayed tables (reference data).

[FIX] Observability

All UI notifications now appear in the Diagnostics panel.

[FIX] UI

Redeploying Data (assembly) no longer adds extra nodes to the navigation tree.

Artifacts

type	location
Infrastructure	kxi-terraform-1.2.0.tgz
Platform	insights-1.2.0.tgz
Operator	kxi-operator-1.2.0.tgz
CLI	kxicli-1.2.0-py3-none-any.whl
ODBC Driver	kodbc 1.2.0
Java SDK	java-sdk 1.2.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Version 1.2.0 requires a custom storageClassName definition

The addition of support for packages introduces a new storageClassName defined as sharedfiles, this is defined by default in any deployments of kdb Insights Enterprise that rely on the KX provided Terraform scripts for deployment but must be added manually by a user as outlined here.

Alternatively, within the deployment configuration as outlined here, modify the storageClassName to an alternative supported RWX storage option for example rookcephfs.

Keycloak 18

The version of Keycloak used within kdb Insights Enterprise has been upgraded from Keycloak 16 to Keycloak 18.

Please follow the Keycloak 18 migration steps to migrate your Keycloak data prior to upgrading kdb Insights Enterprise.

Known Issues

Service GatewayUser InterfaceLicensingSDKsStorage ManagerStream ProcessorAPI GatewayUI Logout

If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below

   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)
The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*
The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.
On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.
When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data

Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.

sm:
  size: 1

Torn-down pipelines that include Database or Stream Writers cannot be directly restarted after having their resources removed. Upon redeploy, data will not be published downstream from the pipeline. A workaround is to rename the pipeline. Resources are removed by checking "Clear Pipeline State" during teardown in the UI, or by directly deleting the Kubernetes PVCs backing the pipeline. If either of these things are done, the pipeline must be renamed before relaunching it.
.qsp.read.fromDatabase cannot be run in a pipeline to query a kdb Insights Enterprise database. The following example code can be used as a workaround to query data (ex. reference data to be used within a pipeline).

onResponse:{[hdr;data]
if[0 <> hdr`ac;
    .sp.log.error ("Query failed host=%s, api=%s, code=%n, error=%s";hdr`gw; hdr`api; hdr`ac; hdr`ai)];
publish data
}
.qsp.onStart {
    h:hopen hsym `$.spenv.getReleaseName[],"-sg-gateway:5050";
    res: neg[h] (`.kxi.getData; (enlist `table)!(enlist `instruments); `onResponse; ()!());
}
.qsp.run .qsp.read.fromCallback[`publish] .qsp.write.toConsole[]

Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:

helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>

Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

After performing a logout action, the user will not be redirected back to the login screen.
In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can navigate re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.1.2

kdb Insights Enterprise 1.1.2 is a patch release that corrects a small number of observability and query issues.

Release Date

2022-07-15

Observability

[FIX] Metrics responses are all now terminated with a newline \n. Absence of the newline caused import issues with some metrics scrapers, for example Azure Monitor.

Explore and Queries

[FIX] The getData API endpoint no longer returns an error when an outputTZ is provided.

[FIX] Queries against the interval tier (idb) around the end-of-day rollover process could return zero results.

Infrastructure

[NEW] The Terraform scripts will now pass the correct platform to the Docker container when running them on Macs

[FIX] The Helm version in the Terraform scripts' Dockerfile has been pinned to a version without AWS compatibility issues.

Artifacts

type	location
Infrastructure	kxi-terraform-1.1.1.tgz
Platform	insights-1.1.2.tgz
Operator	kxi-operator-1.1.0.tgz
CLI	kxicli-0.9.0-py3-none-any.whl
ODBC Driver	kodbc 1.1.0
Java SDK	java-sdk 1.1.0

Upgrade notes

Data loss of UI assembly definitions for versions earlier than 1.1.1

Assembly definitions built from the kdb Insights Enterprise UI in versions earlier than 1.1.1 will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.

To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace.

kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz

To restore, the user should run the following command post-install.

cat kxi-controller-data.tgz |  kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -

kdb Insights Operator Version

As the version of the kdb Insights Operator has not changed from 1.1.0 to 1.1.2, the command for deploying kdb Insights Enterprise is slightly different than in earlier releases. The older kdb Insights Operator version must be explicitly passed to the CLI.

kxi install run --filepath values.yaml --version 1.1.2 --operator-version 1.1.0

Please see the guide to Installing kdb Insights Enterprise or Upgrading kdb Insights Enterprise for more detail.

New functionality and roles

kdb Insights Enterprise 1.1.1 includes new features permissioned by additional roles. If you are upgrading to 1.1.2 from a version earlier than 1.1.1, please see the section about authentication upgrades for more information about how to incorporate realm changes, including new roles, during an upgrade.

Known Issues

Service GatewayUser InterfaceLicensingSDKs

If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below

   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)
Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.
Currently users will be logged out but will not be notified via the UI. If users consistently encounter errors, we suggest they log out and in again.
The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*
The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.
The drop-down list of assemblies in the first Explore tab opened after applying the first kdb Insights Enterprise assembly will take a short while to populate. Try again after a few seconds if nothing appears immediately, or close and re-open the Explore tab to see the list and select an assembly.
Teardown cleanups of Data (assemblies) via the UI will not fully remove PVCs if there are active Explore tabs targeting that assembly. Close these tabs or remove their underlying sandboxes via the REST API to complete the cleanup.

On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.
On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.
When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data

1.1.1

kdb Insights Enterprise 1.1.1 is a patch release that improves usability and stability.

Release Date

2022-06-28

Data (Assemblies)

[FIX] Clearer tooltips for Data status to ensure users can clearly identify when their assemblies are fully deployed.

[FIX] Additional validation has been added to stop re-deployment of Data (assemblies) after schema changes – schema changes on deployed assemblies are not supported in this version of kdb Insights Enterprise.

[FIX] Additional validation has been added for schemas to ensure they can be deployed successfully.

[FIX] The serviceVolume setting is now mapped correctly in Database and Stream

[FIX] A condition where Stream log archiving could fail has been resolved, ensuring that stream log files do not grow beyond the defined limits.

[FIX] Navigation tree issues where schemas could not be opened and deletion failures were not correctly reflected are both resolved.

Pipelines

[FIX] Nodes requiring passwords no longer store them in the pipeline configuration, they must be entered during pipeline deployment.

[FIX] Mandatory fields in nodes are now enforced in pipelines before deployment.

[FIX] Pipeline re-deployments can now optionally clear state (destroying previously persisted pipeline data, but not data already in the database) to allow edits to be made and the pipeline to be restarted from scratch.

[FIX] The error Cannot read properties of undefined (reading '0') which could be thrown when opening multiple pipeline tabs has been resolved.

[FIX] An issue where long code lines in Map nodes could cause browsers to hang has been resolved.

Explore and Queries

[FIX] Session errors Worker has been closed and r.map are no longer thrown.

[FIX] The error No aggregator available when changing assemblies in an Explore session is no longer thrown.

[FIX] Reference data joins which do not include the foreign column or where the foreign column has a different name are now possible.

[FIX] An issue where certain queries’ WHERE clause would work via REST but fail via Explore has been resolved.

[FIX] An issue where certain queries using count could throw the error Error: noupdate: `.Q `pn has been resolved.

Artifacts

type	location
Infrastructure	kxi-terraform-1.1.0.tgz
Platform	insights-1.1.1.tgz
Operator	kxi-operator-1.1.0.tgz
CLI	kxicli-0.9.0-py3-none-any.whl
ODBC Driver	kodbc 1.1.0
Java SDK	java-sdk 1.1.0

Upgrade notes

Data loss of UI assembly definitions

Assembly definitions built from the kdb Insights Enterprise UI will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.

To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace.

kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz

To restore, the user should run the following command post-install.

cat kxi-controller-data.tgz |  kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -

kdb Insights Operator Version

As the version of the kdb Insights Operator has not changed from 1.1.0 to 1.1.1, the command for deploying kdb Insights Enterprise is slightly different than in earlier releases. The older kdb Insights Operator version must be explicitly passed to the CLI.

kxi install run --filepath values.yaml --version 1.1.1 --operator-version 1.1.0

Please see the guide to Installing kdb Insights Enterprise or Upgrading kdb Insights Enterprise for more detail.

New functionality and roles

kdb Insights Enterprise 1.1.1 includes new features permissioned by additional roles. Please see the section about authentication upgrades for more information about how to incorporate realm changes, including new roles, during an upgrade.

Known Issues

Service GatewayUser InterfaceLicensingSDKs

If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below

   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)
Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.
Currently users will be logged out but will not be notified via the UI. If users consistently encounter errors, we suggest they log out and in again.
The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*
The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.
The drop-down list of assemblies in the first Explore tab opened after applying the first Insights assembly will take a short while to populate. Try again after a few seconds if nothing appears immediately, or close and re-open the Explore tab to see the list and select an assembly.

On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.
On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.
When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 if you're upgrading from an earlier version, for notes about:

License changes
Single pod Storage Manager
Data loss of UI assembly definitions

1.1.0

Release for kdb Insights Enterprise.

Release Date

2022-06-07

Features

[NEW] Storage and Query

Late and out-of-order data handling
- Data received out of order is recorded with an appropriate ingest timestamp
- Late data is recorded, irrespective of where its correct destination is in the lifecycle (RDB, IDB, HDB)
- Late data is included by default for all requests, q (getData) and ANSI SQL
- Late data can be filtered out (for example, to see only data present in the system at a particular point in time)
- Performance is not significantly impacted by late data arrival, including for object storage tiers
Reference data can be queried in the Explore Window, in a data pipeline and as part of the getData() API
Support for non time-series reference data in Database
Data in reference tables can be joined to time-series tables in free-form q queries
Improved SQL support for queries (joins and complex queries, except for reference tables)
Compression support, including object storage
Improved resiliency and performance

[NEW] User Interface

Scratchpads
- Data can be pulled into the scratchpad using either sql or getData()
- Freeform q, SQL or Python code can be execute on that data to further filter, aggregate and join data to explore specific records
- Python/q functions or a model can be applied to that data to gain insights
- A model can be fitted using that data
Machine Learning plugins can be added to a Pipeline created from the user interface
More intuitive creation of data ingest pipelines
Observability embedded into the UI (events and logs tab)
Log forwarding to each providers' log aggregator is now enabled by default in the Terraform scripts.

[NEW] Reports

Visualization is available for ingested data using the getData() API

[NEW] Observability

View Database and Pipeline logs and events from the UI
Ability to Export Diagnostics, including the logs, events and state for further diagnosis
Further integration with cloud monitoring tools
Additional performance metrics collected

[NEW] Reliable Transport Stream

Subscribers have the option to override the detail truncation options for the stream logs that they have processed
Improved recovery and performance

Artifacts

type	location
Infrastructure	kxi-terraform-1.1.0.tgz
Platform	insights-1.1.0.tgz
Operator	kxi-operator-1.1.0.tgz
CLI	kxicli-0.9.0-py3-none-any.whl
ODBC Driver	kodbc 1.1.0
Java SDK	java-sdk 1.1.0

Upgrade notes

Data loss of UI assembly definition

Data loss of UI assembly definitions

Assembly definitions built from the kdb Insights Enterprise UI will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.

To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace.

kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz

To restore, the user should run the following command post-install.

cat kxi-controller-data.tgz |  kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -

kdb Insights Enterprise - Previous Release Notes

1.10.1

Release Date 2024-07-24

Fixes

Third-party Dependencies

Artifacts

1.10.0

Release Date 2024-06-20

New Features

1. SQL2 now supports joins

2. Standardized Auditing

3. Environment Variables in Packages

4. kxi-python now supports publish, query and execution of custom APIs

5. Publishing to Reliable Transport (RT) using the CLI

6. Stream Processor supports subscribing to the end of an RT stream

7. Schema Conversion Progress Indicator

8. Natural language descriptions included in getMeta descriptions

9. Cost Reduction Options for Reliable Transport

Improvements

Fixes

Third-party Dependencies

Artifacts

Upgrade/Rollback Notes

Known Issues

Beta Features

Summary

1.9.4

Release Date

Fixes

Third-party Dependencies

Artifacts //! devops to confirm versioning

1.9.3

Release Date

Fixes

Third-party Dependencies

Artifacts

1.9.2

Release Date

Fixes

Third-party Dependencies

Artifacts

1.9.1

Release Date

Improvements

Fixes

Third-party Dependencies

Artifacts

Upgrade Notes

Known Issues

1.9.0

Release Date

New Features

Improvements

Fixes

Third-party Dependencies

Deprecation

Artifacts

Upgrade Notes

Rollback

Known Issues

Deprecation

1.8.2

Release Date

Fixes

Third-party Dependencies

Artifacts

1.8.1

Release Date

Improvements

Fixes

Third-party Dependencies

Artifacts

1.8.0

Release Date

New Features

Improvements

Fixes

Third-party Dependencies

Artifacts

Upgrade notes

8. Natural language descriptions included in `getMeta` descriptions