kdb Insights Enterprise - Release Notes

This page details the release notes for the latest series of kdb Insights Enterprise releases. To find the release notes for an older release, refer to the previous releases page.

1.12.7

Release Date 2025-04-17

Fixes

Expand to see the full details of resolved issues

Database

Resolved an issue where the selectTable API could return an incorrect result if executing an aggregation that included data on disk. For example, a call such as .kxi.selectTabletableagg!(trade;enlist[x]!enlist(avg;price))` could be subject to this defect.

Stream Processor

Resolved a memory leak that occurred when using the MQTT reader.

Security

Various CVEs were remediated as part of this release.

Third-party Dependencies

kdb Insights Enterprise 1.12.7 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.12.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

Type	Nexus location	Downloads Portal location
Enterprise	insights-1.12.7.tgz	insights-1.12.7.tgz
Operator	kxi-operator-1.12.1.tgz	kxi-operator-1.12.1.tgz
CLI	kxicli-1.12.2-py3-none-any.whl	kxicli-1.12.2-py3-none-any.whl
RT C interface	kxi-c-sdk 1.12.0	kxi-c-sdk 1.12.0
RT Python interface	kxi-rtpy-1.12.0	kxi-rtpy-1.12.0
RT Java interface	kxi-java-sdk 1.12.0	kxi-java-sdk 1.12.0
Infrastructure	kxi-terraform-1.12.1.tgz	kxi-terraform-1.12.1.tgz
kxi-management-service	kxi-management-service-0.2.7.tgz	kxi-management-service-0.2.7.tgz
insights-on-k8s	insights-on-k8s-0.2.3.tgz	insights-on-k8s-0.2.3.tgz

1.12.6

Release Date 2025-03-25

Fixes

Expand to see the full details of resolved issues

Database

Schema change performance has been improved to support multi-threaded operations, and to only execute against only the impacted tables.
The database now waits to initialize connection to Reliable Transport until its processes are fully online. Previously, DAPs would sometimes try to connect before fully available, resulting in an excessive number of RT logs being generated and issues with their processing.

Reliable Transport

Resolved an issue in which an incorrect RT_LOG_PATH was set after a restart by qsup.

Stream Processor

Resolved an edge case issue in which the SP would report a session not found error.

Security

Various CVEs were remediated as part of this release.

Third-party Dependencies

kdb Insights Enterprise 1.12.6 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.12.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

Type	Nexus location	Downloads Portal location
Enterprise	insights-1.12.6.tgz	insights-1.12.6.tgz
Operator	kxi-operator-1.12.1.tgz	kxi-operator-1.12.1.tgz
CLI	kxicli-1.12.2-py3-none-any.whl	kxicli-1.12.2-py3-none-any.whl
RT C interface	kxi-c-sdk 1.12.0	kxi-c-sdk 1.12.0
RT Python interface	kxi-rtpy-1.12.0	kxi-rtpy-1.12.0
RT Java interface	kxi-java-sdk 1.12.0	kxi-java-sdk 1.12.0
Infrastructure	kxi-terraform-1.12.1.tgz	kxi-terraform-1.12.1.tgz
kxi-management-service	kxi-management-service-0.2.7.tgz	kxi-management-service-0.2.7.tgz
insights-on-k8s	insights-on-k8s-0.2.3.tgz	insights-on-k8s-0.2.3.tgz

1.12.5

Release Date 2025-03-12

Fixes

Expand to see the full details of resolved issues

Insights Core

Insights 1.12.5 has been updated to run Insights Core 4.1.10, which contains fixes related to the object storage library. Refer to the Insights Core release notes for details of these fixes.

Security

Various CVEs were remediated as part of this release.

Third-party Dependencies

kdb Insights Enterprise 1.12.5 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.12.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

Type	Nexus location	Downloads Portal location
Enterprise	insights-1.12.5.tgz	insights-1.12.5.tgz
Operator	kxi-operator-1.12.1.tgz	kxi-operator-1.12.1.tgz
CLI	kxicli-1.12.2-py3-none-any.whl	kxicli-1.12.2-py3-none-any.whl
RT C interface	kxi-c-sdk 1.12.0	kxi-c-sdk 1.12.0
RT Python interface	kxi-rtpy-1.12.0	kxi-rtpy-1.12.0
RT Java interface	kxi-java-sdk 1.12.0	kxi-java-sdk 1.12.0
Infrastructure	kxi-terraform-1.12.1.tgz	kxi-terraform-1.12.1.tgz
kxi-management-service	kxi-management-service-0.2.7.tgz	kxi-management-service-0.2.7.tgz
insights-on-k8s	insights-on-k8s-0.2.3.tgz	insights-on-k8s-0.2.3.tgz

1.12.4

Release Date 2025-03-05

Improvements

Infrastructure

The Client Terraform scripts now come with an interactive configuration script which supports 3 different infrastructure profiles. Refer to Architecture profile, and the Artifacts section below for details.

Fixes

Expand to see the full details of resolved issues

Database

Executing a .kxi.sql2 or .kxi.getData query and only suppling table name and limit in the arguments caused an out-of-memory error for large databases. Now, the limit is applied (if possible) before unenumerating the data.
Temporary files created by NFS were preventing Storage Manager from deleting directories during on-disk sort. This has been resolved.

Reliable Transport

When subscribing to the latest position in RT, all log files, rather than just the log files with the latest data, were copied to the subscriber node. Now, only the log files required are copied.
RT copies log files from the RT node to the client node. If a log file is truncated by RT on the RT node, it gets truncated on the subscriber node. The subscriber can still truncate the file earlier than RT, but now RT will truncate the file on the client, even if the client has opted out of the RT garbage collection. This mitigates problems that could arise if a client keeps log files longer than RT.

Security

The following CVEs were remediated as part of this release:
- CVE-2024-38541
- CVE-2024-45337
- CVE-2024-38428
- CVE-2024-45337

Third-party Dependencies

kdb Insights Enterprise 1.12.4 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.12.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

Type	Nexus location	Downloads Portal location
Enterprise	insights-1.12.4.tgz	insights-1.12.4.tgz
Operator	kxi-operator-1.12.0.tgz	kxi-operator-1.12.0.tgz
CLI	kxicli-1.12.2-py3-none-any.whl	kxicli-1.12.2-py3-none-any.whl
RT C interface	kxi-c-sdk 1.12.0	kxi-c-sdk 1.12.0
RT Python interface	kxi-rtpy-1.12.0	kxi-rtpy-1.12.0
RT Java interface	kxi-java-sdk 1.12.0	kxi-java-sdk 1.12.0
Infrastructure	kxi-terraform-1.12.1.tgz	kxi-terraform-1.12.1.tgz
kxi-management-service	kxi-management-service-0.2.5.tgz	kxi-management-service-0.2.5.tgz
insights-on-k8s	insights-on-k8s-0.2.3.tgz	insights-on-k8s-0.2.3.tgz

1.12.3

Release Date 2025-02-05

Fixes

Expand to see the full detail of resolved issues here

Database

.kxi.getData would hit a length error when trying to apply more than one filter to a reference table joined via a foreign key.
If executing a .kxi.selectTable query in the RDB for a partitioned table, and the startTS and endTS did not intersect with the purview of the RDB DAP, all data for the table would be returned. This has been addressed so it will now return an empty table if there is no overlap within the purview.

Third-party Dependencies

kdb Insights Enterprise 1.12.3 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.12.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

Type	Nexus location	Downloads Portal location
Enterprise	insights-1.12.3.tgz	insights-1.12.3.tgz
Operator	kxi-operator-1.12.0.tgz	kxi-operator-1.12.0.tgz
CLI	kxicli-1.12.1-py3-none-any.whl	kxicli-1.12.1-py3-none-any.whl
RT C interface	kxi-c-sdk 1.12.0	kxi-c-sdk 1.12.0
RT Python interface	kxi-rtpy-1.12.0	kxi-rtpy-1.12.0
RT Java interface	kxi-java-sdk 1.12.0	kxi-java-sdk 1.12.0
Infrastructure	kxi-terraform-1.12.0.tgz	kxi-terraform-1.12.0.tgz
kxi-management-service	kxi-management-service-0.2.4.tgz	kxi-management-service-0.2.4.tgz
insights-on-k8s	insights-on-k8s-0.2.3.tgz	insights-on-k8s-0.2.3.tgz

1.12.2

Release Date 2025-01-30

Improvements

Database

The helper function .kxi.selectTable now supports the following:
- The limit parameter. When no aggregation is required .kxi.sql2 and .kxi.getData apply the limit parameter closer to the raw data selection, improving performance of the query.
- Filters that use the virtual column i.
When using User-Defined Analytics (UDAs), the userName value is now available when the helper function .kxi.getHeader is called in the DAP. See response header for details on all fields of the response header, and kxi.getHeader for an example.

Fixes

Expand to see the full detail of resolved issues here

Database

Temporary files created by NFS were preventing Storage Manager from deleting directories during post-commit cleanup. This has been resolved through improved handling of .nfs files and changes to how directory deletion is performed.
Resolved an issue in which the Query Environment and general Service Gateway were not registering new services when they were deployed, resulting in queries failing.

Stream Processor

Resolved a socket issue in the Stream Processor when reading from MQTT.

Web Interface

On the Schema Settings page, it was not possible to sort on multiple columns when any column had the parted attribute set. This restriction has been removed, allowing for sorting on multiple columns regardless of attribute.
On the Pipelines page, there was previously no indication when a pipeline only partially succeeded in its operations. Now, pipelines report partial status, making confirmation or debugging clearer.
On the Databases page, the Pipelines box previously gave the impression that you could delete the pipeline by using the backspace/delete key in that field. However, this action does not actually remove the pipeline. The ability to use backspace/delete on a pipeline name in this field has been disabled.

Entitlements

If using entitlements, upgrading from 1.12.0 to 1.12.1 would fail as a result of a duplicate environment variable KXI_ENTITILEMENT_SERVICE in the Observability service. This has been resolved in 1.12.2, and for previous versions, a workaround is available, in which the Observability service can be deleted prior to performing your upgrade. This could be done by executing kubectl delete deploy/insights-kxi-obs-srv.

Third-party Dependencies

kdb Insights Enterprise 1.12.2 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.12.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type	Nexus location	Downloads Portal location
Enterprise	insights-1.12.2.tgz	insights-1.12.2.tgz
Operator	kxi-operator-1.12.0.tgz	kxi-operator-1.12.0.tgz
CLI	kxicli-1.12.1-py3-none-any.whl	kxicli-1.12.1-py3-none-any.whl
RT C interface	kxi-c-sdk 1.12.0	kxi-c-sdk 1.12.0
RT Python interface	kxi-rtpy-1.12.0	kxi-rtpy-1.12.0
RT Java interface	kxi-java-sdk 1.12.0	kxi-java-sdk 1.12.0
Infrastructure	kxi-terraform-1.12.0.tgz	kxi-terraform-1.12.0.tgz
kxi-management-service	kxi-management-service-0.2.4.tgz	kxi-management-service-0.2.4.tgz
insights-on-k8s	insights-on-k8s-0.2.3.tgz	insights-on-k8s-0.2.3.tgz

Upgrade/Rollback

See Upgrade/Rollback Notes for 1.12 for considerations.

1.12.1

Release Date 2025-01-21

Improvements

Database

Improvements have been made in Storage Manager to reduce the time spent on migrating data across tiers (when compression settings across tiers are the same).

A new optional environment variable KXI_SKIP_HDB_SIZE has been introduced to bypass the HDB size calculation during startup. When enabled, this variable skips the calculation of HDB sizes during SM initialization which is beneficial for systems with large HDBs, where the size calculation can be time-consuming. See Environment Variables for details.

When using User-Defined Analytics (UDAs), the userID value is now available when the helper function .kxi.getHeader is called in the DAP. See response header for details on all fields of the response header, and kxi.getHeader for an example.

Packaging

The kxi package checkpoint command no longer creates a package zip by default. This option is still available if needed. See -- kxi in the kxi package checkpoint documentation for details.

Python Pipelines

Python pipelines now support reading from databases using User-Defined Analytics (UDAs). For more information, refer to the Python readers documentation.

Fixes

Expand to see the full detail of resolved issues here

Database

On the Schema Settings page, there was no default or minimum block size applied. Depending on how this was set, this could cause adverse effects on the database. A new appropriate minimum of 50,000 rows has now been set, with no maximum. See Performance for more details on block size.
Resolved an issue where late data added to an existing HDB table partition could create duplicate records at EOD write-down if the memory available to the Storage Manager process is very small.
Resolved an issue where the KXI_SM_HDB_SIZE metric caused delays in SM initialization for systems with large historical databases (HDBs).
Storage Manager now catches attempted column type conversions that aren't supported and provides a more helpful log message before shutting down.
When executing a batch ingest that spans multiple days, the Storage Manager now performs RAM garbage collection between dates. This avoids the potential for out-of-memory failures.

Readers

The MQTT reader now obeys the $no_proxy environment variable when making connections. See deploying behind a proxy server for details on this feature.

Web Interface

Used and available Query, Writedown, and Stream resources were sometimes not being populated in the Database Overview page. This has been resolved.
When issuing a query (in the Query or Scratchpad panels) with the console tab selected, memory usage at the bottom of the screen was not updated if you selected either the table or visual tab. This has been resolved.

Packaging

Two databases with the same name in different packages could not be distinguished in the list of databases in the Database node editor. The package name has been appended to the list to ensure the databases can be identified correctly.
If you deleted a package that contained entities (pipelines, databases, etc.), and those entities were open in their own tabs, the web interface would throw a 404 error when trying to view them in their respective tabs. This has been resolved such that any open tabs for entities in the package are closed.
When tearing down a package in the web interface, the option to Clean up entities after teardown erroneously stated that existing data would not be deleted with this operation. This has been corrected to Warning: Cleaning up entities will delete all data associated with a database and all pipeline checkpoints and user states. If running 1.12.0, the incorrect error message will still be seen.

Third-party Dependencies

kdb Insights Enterprise 1.12.1 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.12.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type	Nexus location	Downloads Portal location
Enterprise	insights-1.12.1.tgz	insights-1.12.1.tgz
Operator	kxi-operator-1.12.0.tgz	kxi-operator-1.12.0.tgz
CLI	kxicli-1.12.1-py3-none-any.whl	kxicli-1.12.1-py3-none-any.whl
RT C interface	kxi-c-sdk 1.12.0	kxi-c-sdk 1.12.0
RT Python interface	kxi-rtpy-1.12.0	kxi-rtpy-1.12.0
RT Java interface	kxi-java-sdk 1.12.0	kxi-java-sdk 1.12.0
Infrastructure	kxi-terraform-1.12.0.tgz	kxi-terraform-1.12.0.tgz
kxi-management-service	kxi-management-service-0.2.4.tgz	kxi-management-service-0.2.4.tgz
insights-on-k8s	insights-on-k8s-0.2.3.tgz	insights-on-k8s-0.2.3.tgz

Upgrade/Rollback

See Upgrade/Rollback Notes for 1.12 for considerations.

Known Issues

Expand to see the list of known issues here

User-Defined Analytics (UDAs)

If a UDA has a table hardcoded in the query API, then it will throw the following error Error: Querying database using (UDA) raised - Unknown API: .uda.noParamAPI in the UDA Query tab.

Entitlements

If a user is a member of multiple user groups, their access level is based solely on the first group processed in the list of groups when checking entitlements.

1.12.0

Release Date 2024-12-11

kdb Insights Enterprise 1.12 introduces a number of new features and improvements, including; an enhanced data import and ingest experience, the ability to assign row level entitlements to tables (beta), enhancements to Views, our new Power BI connector and more. Read on to find out more!

New Features

Batch Ingest Features & Enhancements
Initial Database Import
Performance Tuning for Schemas
Row Level Entitlements (BETA)
Natively define Python User-Defined Analytics (UDAs)
Support calling UDAs from SP Pipeline Builder and the Query Tab
Power BI Certified Connector
New Filtering Options for Views
VSCode Users - Populate your scratchpad from q code
Second Y-Axis in Views

1. Batch Ingest Features & Enhancements

This release introduces a new version of the database writer, which improves the batch ingest experience when using direct write. The new database writer includes:

Support for partitioned pipelines, allowing improved performance using parallelization. When maxWorkers is set to more than 1, the Stream Processor (SP) scales up to multiple workers, each taking a shard of the workload and processing it in parallel. This means a much faster processing time.
Support for unbounded readers, allowing pipelines to read continuous streams of data but still use direct write to the database. Examples of unbounded use cases include file readers with file watching enabled, stream readers (Kafka, Insights Streams, MQTT), callback readers, or upload readers. This support removes the restrictions and limitations around the support of these use cases.
Monitoring of database ingest sessions using a new API or the web interface. This allows you to monitor the state of ingest sessions by returning a list of them and their various states: pending, queued, progressing, completed or errored.

An overview of the database ingest functionality is available here, and the API details can be found here: Python and q.

A walkthrough is also available for guidance.

2. Initial Database Import Enhancements

You can import your existing kdb+ database into kdb Insights Enterprise to seed your empty database with data. This release improves upon the existing the process, and also provides:

Embedded validation to ensure the alignment of the schemas and the appropriate attributes
Graceful handling of failures with a supported way to re-try/re-start ingest if it does fail for some reason

Read the quickstart guide for a walkthrough of using the initial import feature. The full import process can be followed here.

3. Performance Tuning for Schemas

kdb Insights Enterprise now provides in-app guidance when configuring a schema, helping new kdb+ users define appropriate table definitions and attibrutes to achieve a more optimal schema and better performance. This assists you to determine:

What table type to use for different types of data
What attribute(s) to set for a table for optimal performance
Setting an appropriate column as the basis for partitioning a table
When attributes have not been set

Read the Performance Tuning documentation for more details.

4. Row Level Entitlements (BETA)

You can now manage user access to data at the row level. Extending our existing database-level query entitlements, you can now apply policies to manage group’s entitlements to rows of data in each table in your database.

See Row Level Entitlements for more details.

5. Natively define Python User-Defined Analytics (UDAs)

You can now develop your UDAs natively in Python and maintain and deploy these as part of a package. Read the Python API documentation regarding UDAs for more details.

6. Support calling UDAs from Pipelines and the Query Tab

UDAs can now be queried from pipelines using the database reader, or from the Query tab. Read query using UDAs and the UDAs section of the Query APIs documentation for how to take advantage of this feature.

7. Power BI Certified Connector

Together with KX, Microsoft have certified and released the kdb Insights Enterprise Power Query Connector. Built using our REST interface, the connector supports Power BI Import Mode to allow you to import a subset of your kdb+ data from kdb Insights Enterprise to visualize in Power BI. This feature leverages Insights' versatile getData API, allowing you to filter, group by and aggregate your data before importing the results into Power BI for visualization.

8. New Filtering Options for Streaming to Views

There are now additional filtering capabilities for streaming subscribers in Views, allowing you to:

Filter on columns with data types other than sym
Filter on a list of columns

Read the subscriber documentation for more information. This walkthrough also has helpful examples.

If using a streaming data source, there are some limitations to be aware of. Read this walkthrough and the details listed in the Streaming tab.

9. Query the database using the q API

You can now query the database using a q API from both the query tab and the kdb+ VSCode Extension. For more details, read the Query API documentation.

10. Second y-axis in the Query Window

Within the Query window, you can now add a second y-axis on the visualization tab to better display different datasets with differences in scale or different chart types. Refer to the documentation for details.

Improvements

Updates to Package Administration

Packages can now be administered from the kdb Insights Enterprise web interface, providing more flexibility when creating, updating and viewing packages. Read the packaging documentation for more details.
Additionally, this release introduces an updated set of CLI commands for package management. kxi pm is the new and improved command set for pushing, deploying, and tearing down packages. In the CLI, run kxi pm --help for the updated set of commands. This quickstart also illustrates the use of kxi pm.

Note that you can still use the kxi package command for building of packages on your local machine, and for remote (on-server) operations as well. However, this is being deprecated as of 1.12.0 and replaced with kxi pm for both types of operations. Read Deprecations below for specifics.

CLI

The CLI user experience has been streamlined to simplify managing entitlements. Using kxi entitlement manage, you can now easily perform these operations:
- Add groups to entitlements
- Remove groups from entitlements
- Setting Access levels to groups

Read the CLI entitlements documentation for details.

Audit Logging

The following operations now have audit logging to offer traceability of key system operations:

CLI - adding, modifying or deleting an entitlement now generates audit logs to indicate the details of the user, time, and specifics of the action.
Packages - creating, saving, deploying, tearing down or deleting generates audit logs, including the name of the package with a correlator ID.
Queries - the entirety of the query path now provides an audit-id that makes tracing your query across processes quick and easy.
Stream Processor - Correlation IDs are now logged by the audit framework on successful and failed SP upload requests.

Entitlements

Logs now comply with entitlements and restrict user access to logs they are not entitled to see. In the Diagnostics page of the web interface, users cannot see logs for a deployed database(s)/pipeline(s) unless they are entitled to the related package.

Access to logs is now restricted based on entitlements, but pipelines deployed using UI will still be open to everyone.

New SQL Operators

SQL2 now supports the following new operators:
- reverse
- char_length
- character_length
- bit_length
- length
- octet_length

Parquet Reader

A local file watcher has been added for the Parquet reader. Read the q and Python documentation for details.
The Parquet reader now allows you to specify the file path as a symbol.

Stream Processor

The Python API has been updated to include .qsp.track functionality that the q API already supports. Read the track page for details.

Scratchpads

Any running scratchpad pods are now torn down when Insights is uninstalled.
You may now enable Beta Features in the scratchpad or in Quick Test by simply setting an Environment Variable value. To enable beta features, set the environment variable KXI_SP_BETA_FEATURES to true.

Streaming to Views

The Views JSON filter now supports a list of values. For example:
```
{"sym": ["EUR/USD","GBP/USD"]}
```

Database Column Re-sorting

It is now possible to change the defined sort column of a table(s) for an existing database. This is an expensive, long-running operation for existing, large-scale databases. This should be considered before execution. For this reason, it is not enabled by default. To use, set the environment variable KXI_SM_ALLOW_CHANGE_SORT_COLS=1 in Storage Manager and then follow the steps for modifying a schema.

Fixes

Expand to see the full detail of resolved issues here

Database

Resolved an issue in which Storage Manager's RAM requirements increased for batch ingest operations. This has been returned to resource levels in line with release 1.10.
Previously, a failed end-of-day writedown (EOD) was not retried by the Storage Manager, causing a misaligned error on the next EOD.
Fixed an issue when the Database was not being identified as having unsaved changes when primary keys were changed.
Fixed an issue where DAP code assumed the values of assembly mount names was one of "RDB", "IDB", or "HDB". This has been resolved to allow for mount names of any description.
Resolved an issue where a SQL2 query would have produced a type error when using a GROUP BY statement with an ORDER BY of a subset of the fields with no aggregation. For example, select sym,realTime from trade group by sym, realTime order by sym.
Resolved an issue where a SQL2 query would have produced a type error when using a GROUP BY statement with a DISTINCT and no other aggregation.
Resolved an issue where a query for an empty date could fail when late data is turned on.
Previously, if a Stream Processor published a message to a subscriber node that contained a null float, and the WebSocket client reads the messages in JSON, the Service Gateway would error with NaN is not a valid double value as per JSON specification error. This has been resolved.

Entitlements

The Viewer role was previously unable to run UDAs. This would result in Views using a custom API to fail with a permissions error and not be able to view the data. This has been resolved.
The Entitlement Service was not starting when a Proxy was enabled. This has been resolved.

Packages

Previously, when entitlements were turned on, you could not load a package in the scratchpad using Python, or use a Python UDF. This has been resolved.
Resolved a previously-reported known issue that occurred when you removed a pipeline from a database, and packaging preserved this pipeline in a separate package to ensure it was not permanently deleted. However, the pipeline was still available in the web interface. Any re-pushing of the package from the CLI would cause the pipeline to exist in two packages, with the web interface serving the wrong version.

Views

Resolved formatting issues that would occur when a user with no write access was on a Views tab.
On the Views API form, a parameter of type string was displayed incorrectly as list. This has been resolved.

Readers

An issue where the user could not aggregate data as part of the DB Reader in the web interface, and in certain cases when using the API, has been fixed. This also includes that fill is only sent if explicitly specified.
Using a Parquet reader with a glob pattern would error when trying to find or read a file. This has been resolved.
Fixed an edge-case issue where a length error would occur Failed to initialize worker, worker name=[sample name] error=length when using glob patterns with file names.
Users will no longer see a warning icon in the web interface when deploying pipelines that involve readers and writers without intra-day write-down support. Instead, an informational message will be logged. This is to be expected in this scenario and there is no action required from the user.
The discrepancy in how AWS and Azure Cloud readers retained an extra newline character when reading in text mode without chunking, while the Google Cloud reader correctly did not, has now been resolved.
When establishing a secret used for GCP authentication for a Parquet reader, a pipeline would fail with a Worker failed to start: Unable to locate credentials secret error.

File Watchers

Passing in a dict for the watch parameter was not working correctly. Furthermore, use of the additional options the web interface provides when Use Watching tick box meant a dict was used, and therefore also failed. This has been resolved.

Pipelines

In the web interface, using the "Copy pipeline" button on the Database Details page would result in the pipeline being copied, but with a default graph pipeline instead of the actual pipeline code. This has been resolved.

Third-party Dependencies

kdb Insights Enterprise 1.12.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.12.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type	Nexus location	Downloads Portal location
Enterprise	insights-1.12.0.tgz	insights-1.12.0.tgz
Operator	kxi-operator-1.12.0.tgz	kxi-operator-1.12.0.tgz
CLI	kxicli-1.12.0-py3-none-any.whl	kxicli-1.12.0-py3-none-any.whl
RT C interface	kxi-c-sdk 1.12.0	kxi-c-sdk 1.12.0
RT Python interface	kxi-rtpy-1.12.0	kxi-rtpy-1.12.0
RT Java interface	kxi-java-sdk 1.12.0	kxi-java-sdk 1.12.0
Infrastructure	kxi-terraform-1.12.0.tgz	kxi-terraform-1.12.0.tgz
kxi-management-service	kxi-management-service-0.2.4.tgz	kxi-management-service-0.2.4.tgz
insights-on-k8s	insights-on-k8s-0.2.3.tgz	insights-on-k8s-0.2.3.tgz

Upgrading to kdb Insights Enterprise 1.12

Before upgrading to 1.12 it is necessary to upgrade to the 1.12 CLI.

If upgrading to 1.11 from a prior version before to upgrading to 1.12, you would need to upgrade your CLI and ensure an encryption secret has been created first. See below for details.

You can create an encryption secret by running the kxi install create-encryption-key command as follows:

kxi install create-encryption-key

By default a secret called kxi-encryption-secret is created. If you wish to use a different name for the secret you can either:

Specify the secret name when running the command and update your installation configuration file to use it:
1. Run the following, replacing dare-secret-value with your chosen secret name:
```
kxi install create-encryption-key --database-encryption-secret dare-secret-value
```
2. Merge the text below into your existing installation configuration file, replacing dare-secret-value with your chosen secret name:
```
global:
encryption:
    auth:
    existingSecret: dare-secret-value
```
Use kxi install setup and merge the output with your existing configuration file:
1. Run kxi install setup which will automatically generate an encryption secret if one doesn't already exist, prompting you for a secret name.
2. Merge the configuration output file from this command with your existing configuration file by referring to applying configuration changes.

With either method of generating an encryption secret, if a secret already exists with the name specified, it is not overwritten. To replace a secret, you must delete the existing secret and then a create a new one.

Read ecryption secret for more information on secrets and how to administer them.

Upgrade/Rollback Notes for 1.12

If performing a rollback from 1.12.x to any earlier version, assemblies must not be running prior to initiating the rollback to ensure no data is lost. See Upgrading for how to perform upgrades/rollbacks and the available commands to use. Please see the following specific instructions per version:

For rollback from 1.12.x to 1.12.0:

Rollback kdb Insights Enterprise per the rollback instructions referenced above.

For rollback from 1.12.x to 1.11.x:

Teardown your package(s).
Run the kxi-db-rollback chart with the following command for each package in turn, substituting the desired values:

helm install rollback kx-insights/kxi-db-rollback --version="[current Insights version]" --set rollback.assembly="[your-package-name]" --set rollback.version="[Insights version to rollback to]" -f[your values.yaml file]

Uninstall the rollback chart between each invocation of the kxi-db-rollback chart:

helm uninstall rollback

Repeat this step until all packages have been processed (including any which weren't deployed).

Rollback kdb Insights Enterprise per the rollback instructions referenced above.
Redeploy the packages which you tore down.

For rollback from 1.12.x to 1.10.x:

Teardown your package(s).
Run the kxi-db-rollback chart with the following command for each package in turn, substituting the desired values:

helm install rollback kx-insights/kxi-db-rollback --version="[current Insights version]" --set rollback.assembly="[your-package-name]" --set rollback.version="[Insights version to rollback to]" --set rollback.currentVersion="[current Insights version]" -f[your values.yaml file]

Uninstall the rollback chart between each invocation of the kxi-db-rollback chart:

helm uninstall rollback

Repeat this step until all packages have been processed (including any which weren't deployed).

Rollback kdb Insights Enterprise per the rollback instructions referenced above.
Redeploy the packages which you tore down.

Additional Upgrade Notes

Draft Databases & Pipelines

As of 1.11, draft databases and pipelines that have never been saved will not be available for use. If you have draft databases or pipelines stored in your browser, they will need to be deleted. Alternatively, save your draft databases and pipelines prior to upgrade. Please be aware of this if you are upgrading to 1.11 before moving to 1.12.

Known Issues

Expand to see the list of known issues here

Entitlements

If a user is a member of multiple user groups, their access level is based solely on the first group processed in the list of groups when checking entitlements.

Cleaning up entities

WARNING: When tearing down a package in the web interface, the option to Clean up entities after teardown WILL DELETE ALL DATA, despite the note in the pop-up window stating the opposite: Data that has already been written down will NOT be affected. To avoid data deletion, DO NOT select the Clean up entities option. If a partial clean-up is required, use the relevant kubectl CLI commands instead.

PostGreSQL & Keycloak Resiliency

A transient issue can occur in which Keycloak is unable to re-establish a connection to PostgreSQL after it recovers from an outage. This can result in getData being unable to serve queries. To work around this, restart PostgreSQL.

Upgrades & Rollbacks

Users are prompted to tear down assemblies upon rollback. The rollback should exit if the user responds ‘n' to this prompt. Instead, the rollback continues.

Deprecations

The kxi package commands for remote operations has been deprecated as of the 1.12 release. kxi pm offers full functional equivalency.
- pull
- push
- remote-list
- remote-remove
- remote-convert
- remote-config
- deploy
- teardown
The RT Bridge has been deprecated as of the 1.12 release. The rt.qpk now offers full functional equivalency to the RT bridge so it is recommended to migrate to it prior to RT Bridge being removed.

Summary

We hope you find some useful features that optimize your kdb Insights Enterprise experience. Try them out and email our Support Team if you need any help.

We look forward to bring you even bigger features in kdb Insights Enterprise 1.13 coming soon!