kdb Insights Enterprise - Release Notes

This page details the release notes for the latest series of kdb Insights Enterprise releases. To find the release notes for an older release, refer to the previous releases page.

1.12.1

Release Date 2025-01-21

Improvements

Database

• Improvements have been made in Storage Manager to reduce the time spent on migrating data across tiers (when compression settings across tiers are the same).

• A new optional environment variable KXI_SKIP_HDB_SIZE has been introduced to bypass the HDB size calculation during startup. When enabled, this variable skips the calculation of HDB sizes during SM initialization which is beneficial for systems with large HDBs, where the size calculation can be time-consuming. See Environment Variables for details.

• When using User-Defined Analytics (UDAs), the userID value is now available when the helper function .kxi.getHeader is called in the DAP. See response header for details on all fields of the response header, and kxi.getHeader for an example.

Packaging

• The kxi package checkpoint command no longer creates a package zip by default. This option is still available if needed. See -- kxi in the kxi package checkpoint documentation for details.

Python Pipelines

• Python pipelines now support reading from databases using User-Defined Analytics (UDAs). For more information, refer to the Python readers documentation.

Fixes

Expand to see the full detail of resolved issues here

Database

• On the Schema Settings page, there was no default or minimum block size applied. Depending on how this was set, this could cause adverse effects on the database. A new appropriate minimum of 50,000 rows has now been set, with no maximum. See Performance for more details on block size.

• Resolved an issue where late data added to an existing HDB table partition could create duplicate records at EOD write-down if the memory available to the Storage Manager process is very small.

• Resolved an issue where the KXI_SM_HDB_SIZE metric caused delays in SM initialization for systems with large historical databases (HDBs).

• Storage Manager now catches attempted column type conversions that aren't supported and provides a more helpful log message before shutting down.

• When executing a batch ingest that spans multiple days, the Storage Manager now performs RAM garbage collection between dates. This avoids the potential for out-of-memory failures.

Readers

• The MQTT reader now obeys the $no_proxy environment variable when making connections. See deploying behind a proxy server for details on this feature.

Web Interface

• Used and available Query, Writedown, and Stream resources were sometimes not being populated in the Database Overview page. This has been resolved.

• When issuing a query (in the Query or Scratchpad panels) with the console tab selected, memory usage at the bottom of the screen was not updated if you selected either the table or visual tab. This has been resolved.

Packaging

• Two databases with the same name in different packages could not be distinguished in the list of databases in the Database node editor. The package name has been appended to the list to ensure the databases can be identified correctly.

• If you deleted a package that contained entities (pipelines, databases, etc.), and those entities were open in their own tabs, the web interface would throw a 404 error when trying to view them in their respective tabs. This has been resolved such that any open tabs for entities in the package are closed.

• When tearing down a package in the web interface, the option to Clean up entities after teardown erronesouly stated that existing data would not be deleted with this operation. This has been corrected to Warning: Cleaning up entities will delete all data associated with a database and all pipeline checkpoints and user states. If running 1.12.0, the incorrect error message will still be seen.

Third-party Dependencies

kdb Insights Enterprise 1.12.1 standalone install supports the following versions of third-party dependencies:

• cert-manager - 1.16.1
• nginx-ingress - 4.11.3
• rook-ceph - 1.15.5
• Kubernetes - 1.29

These versions are used in the 1.12.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

• OpenShift
• OpenSource K8s
• EKS, AKS, and GCP

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type	Nexus location	Downloads Portal location
Enterprise	insights-1.12.1.tgz	insights-1.12.1.tgz
Operator	kxi-operator-1.12.0.tgz	kxi-operator-1.12.0.tgz
CLI	kxicli-1.12.1-py3-none-any.whl	kxicli-1.12.1-py3-none-any.whl
RT C interface	kxi-c-sdk 1.12.0	kxi-c-sdk 1.12.0
RT Python interface	kxi-rtpy-1.12.0	kxi-rtpy-1.12.0
RT Java interface	kxi-java-sdk 1.12.0	kxi-java-sdk 1.12.0
Infrastructure	kxi-terraform-1.12.0.tgz	kxi-terraform-1.12.0.tgz
kxi-management-service	kxi-management-service-0.2.4.tgz	kxi-management-service-0.2.4.tgz
insights-on-k8s	insights-on-k8s-0.2.3.tgz	insights-on-k8s-0.2.3.tgz

Upgrade/Rollback Notes

If performing a rollback from 1.12.x to any earlier version, assemblies must not be running prior to initiating the rollback to ensure no data is lost. See Upgrading for how to perform upgrades/rollbacks and the available commands to use. Please see the following specific instructions per version:

For rollback from 1.12.x to 1.12.0:

1. Rollback kdb Insights Enterprise per the rollback instructions referenced above.

For rollback from 1.12.x to a version earlier than 1.12.0:

1. Teardown your package(s).
2. Run the kxi-db-rollback chart with the following command for each package in turn, substituting the desired values:

helm install rollback kx-insights/kxi-db-rollback --version="[current Insights version]" --set rollback.assembly="[your-package-name]" --set rollback.version="[Insights version to rollback to]" -f[your values.yaml file]

Uninstall the rollback chart between each invocation of the kxi-db-rollback chart:

helm uninstall rollback

Repeat this step until all packages have been processed (including any which weren't deployed).

1. Rollback kdb Insights Enterprise per the rollback instructions referenced above.
2. Redeploy the packages which you tore down.

Known Issues

Expand to see the list of known issues here

User-Defined Analytics (UDAs)

• If a UDA has a table hardcoded in the query API, then it will throw the following error Error: Querying database using (UDA) raised - Unknown API: .uda.noParamAPI in the UDA Query tab.

Entitlements

• If a user is a member of multiple user groups, their access level is based solely on the first group processed in the list of groups when checking entitlements.

1.12.0

Release Date 2024-12-11

kdb Insights Enterprise 1.12 introduces a number of new features and improvements, including; an enhanced data import and ingest experience, the ability to assign row level entitlements to tables (beta), enhancements to Views, our new Power BI connector and more. Read on to find out more!

New Features

1. Batch Ingest Features & Enhancements
2. Initial Database Import
3. Performance Tuning for Schemas
4. Row Level Entitlements (BETA)
5. Natively define Python User-Defined Analytics (UDAs)
6. Support calling UDAs from SP Pipeline Builder and the Query Tab
7. Power BI Certified Connector
8. New Filtering Options for Views
9. VSCode Users - Populate your scratchpad from q code
10. Second Y-Axis in Views

1. Batch Ingest Features & Enhancements

This release introduces a new version of the database writer, which improves the batch ingest experience when using direct write. The new database writer includes:

• Support for partitioned pipelines, allowing improved performance using parallelization. When maxWorkers is set to more than 1, the Stream Processor (SP) scales up to multiple workers, each taking a shard of the workload and processing it in parallel. This means a much faster processing time.
• Support for unbounded readers, allowing pipelines to read continuous streams of data but still use direct write to the database. Examples of unbounded use cases include file readers with file watching enabled, stream readers (Kafka, Insights Streams, MQTT), callback readers, or upload readers. This support removes the restrictions and limitations around the support of these use cases.
• Monitoring of database ingest sessions using a new API or the web interface. This allows you to monitor the state of ingest sessions by returning a list of them and their various states: pending, queued, progressing, completed or errored.

An overview of the database ingest functionality is available here, and the API details can be found here: Python and q.

A walkthrough is also available for guidance.

2. Initial Database Import Enhancements

You can import your existing kdb+ database into kdb Insights Enterprise to seed your empty database with data. This release improves upon the existing the process, and also provides:

• Embedded validation to ensure the alignment of the schemas and the appropriate attributes
• Graceful handling of failures with a supported way to re-try/re-start ingest if it does fail for some reason

Read the quickstart guide for a walkthrough of using the initial import feature. The full import process can be followed here.

3. Performance Tuning for Schemas

kdb Insights Enterprise now provides in-app guidance when configuring a schema, helping new kdb+ users define appropriate table definitions and attibrutes to achieve a more optimal schema and better performance. This assists you to determine:

• What table type to use for different types of data
• What attribute(s) to set for a table for optimal performance
• Setting an appropriate column as the basis for partitioning a table
• When attributes have not been set

Read the Performance Tuning documentation for more details.

4. Row Level Entitlements (BETA)

You can now manage user access to data at the row level. Extending our existing database-level query entitlements, you can now apply policies to manage group’s entitlements to rows of data in each table in your database.

See Row Level Entitlements for more details.

5. Natively define Python User-Defined Analytics (UDAs)

You can now develop your UDAs natively in Python and maintain and deploy these as part of a package. Read the Python API documentation regarding UDAs for more details.

6. Support calling UDAs from Pipelines and the Query Tab

UDAs can now be queried from pipelines using the database reader, or from the Query tab. Read query using UDAs and the UDAs section of the Query APIs documentation for how to take advantage of this feature.

7. Power BI Certified Connector

Together with KX, Microsoft have certified and released the kdb Insights Enterprise Power Query Connector. Built using our REST interface, the connector supports Power BI Import Mode to allow you to import a subset of your kdb+ data from kdb Insights Enterprise to visualize in Power BI. This feature leverages Insights' versatile getData API, allowing you to filter, group by and aggregate your data before importing the results into Power BI for visualization.

8. New Filtering Options for Streaming to Views

There are now additional filtering capabilities for streaming subscribers in Views, allowing you to:

• Filter on columns with data types other than sym
• Filter on a list of columns

Read the subscriber documentation for more information. This walkthrough also has helpful examples.

If using a streaming data source, there are some limitations to be aware of. Read this walkthrough and the details listed in the Streaming tab.

9. Query the database using the q API

You can now query the database using a q API from both the query tab and the kdb+ VSCode Extension. For more details, read the Query API documentation.

10. Second y-axis in the Query Window

Within the Query window, you can now add a second y-axis on the visualization tab to better display different datasets with differences in scale or different chart types. Refer to the documentation for details.

Improvements

Updates to Package Administration

• Packages can now be administered from the kdb Insights Enterprise web interface, providing more flexibility when creating, updating and viewing packages. Read the packaging documentation for more details.

• Additionally, this release introduces an updated set of CLI commands for package management. kxi pm is the new and improved command set for pushing, deploying, and tearing down packages. In the CLI, run kxi pm --help for the updated set of commands. This quickstart also illustrates the use of kxi pm.

Note that you can still use the kxi package command for building of packages on your local machine, and for remote (on-server) operations as well. However, this is being deprecated as of 1.12.0 and replaced with kxi pm for both types of operations. Read Deprecations below for specifics.

CLI

• The CLI user experience has been streamlined to simplify managing entitlements. Using kxi entitlement manage, you can now easily perform these operations:
  • Add groups to entitlements
  • Remove groups from entitlements
  • Setting Access levels to groups

Read the CLI entitlements documentation for details.

Audit Logging

The following operations now have audit logging to offer traceability of key system operations:

• CLI - adding, modifying or deleting an entitlement now generates audit logs to indicate the details of the user, time, and specifics of the action.
• Packages - creating, saving, deploying, tearing down or deleting generates audit logs, including the name of the package with a correlator ID.
• Queries - the entirety of the query path now provides an audit-id that makes tracing your query across processes quick and easy.
• Stream Processor - Correlation IDs are now logged by the audit framework on successful and failed SP upload requests.

Entitlements

• Logs now comply with entitlements and restrict user access to logs they are not entitled to see. In the Diagnostics page of the web interface, users cannot see logs for a deployed database(s)/pipeline(s) unless they are entitled to the related package.

Access to logs is now restricted based on entitlements, but pipelines deployed using UI will still be open to everyone.

New SQL Operators

• SQL2 now supports the following new operators:
  • reverse
  • char_length
  • character_length
  • bit_length
  • length
  • octet_length

Parquet Reader

• A local file watcher has been added for the Parquet reader. Read the q and Python documentation for details.

• The Parquet reader now allows you to specify the file path as a symbol.

Stream Processor

• The Python API has been updated to include .qsp.track functionality that the q API already supports. Read the track page for details.

Scratchpads

• Any running scratchpad pods are now torn down when Insights is uninstalled.

• You may now enable Beta Features in the scratchpad or in Quick Test by simply setting an Environment Variable value. To enable beta features, set the environment variable KXI_SP_BETA_FEATURES to true.

Streaming to Views

• The Views JSON filter now supports a list of values. For example:

  {"sym": ["EUR/USD","GBP/USD"]}
  

Database Column Re-sorting

• It is now possible to change the defined sort column of a table(s) for an existing database. This is an expensive, long-running operation for existing, large-scale databases. This should be considered before execution. For this reason, it is not enabled by default. To use, set the environment variable KXI_SM_ALLOW_CHANGE_SORT_COLS=1 in Storage Manager and then follow the steps for modifying a schema.

Fixes

Expand to see the full detail of resolved issues here

Database

• Resolved an issue in which Storage Manager's RAM requirements increased for batch ingest operations. This has been returned to resource levels in line with release 1.10.

• Previously, a failed end-of-day writedown (EOD) was not retried by the Storage Manager, causing a misaligned error on the next EOD.

• Fixed an issue when the Database was not being identified as having unsaved changes when primary keys were changed.

• Fixed an issue where DAP code assumed the values of assembly mount names was one of "RDB", "IDB", or "HDB". This has been resolved to allow for mount names of any description.

• Resolved an issue where a SQL2 query would have produced a type error when using a GROUP BY statement with an ORDER BY of a subset of the fields with no aggregation. For example, select sym,realTime from trade group by sym, realTime order by sym.

• Resolved an issue where a SQL2 query would have produced a type error when using a GROUP BY statement with a DISTINCT and no other aggregation.

• Resolved an issue where a query for an empty date could fail when late data is turned on.

• Previously, if a Stream Processor published a message to a subscriber node that contained a null float, and the WebSocket client reads the messages in JSON, the Service Gateway would error with NaN is not a valid double value as per JSON specification error. This has been resolved.

Entitlements

• The Viewer role was previously unable to run UDAs. This would result in Views using a custom API to fail with a permissions error and not be able to view the data. This has been resolved.

• The Entitlement Service was not starting when a Proxy was enabled. This has been resolved.

Packages

• Previously, when entitlements were turned on, you could not load a package in the scratchpad using Python, or use a Python UDF. This has been resolved.

• Resolved a previously-reported known issue that occurred when you removed a pipeline from a database, and packaging preserved this pipeline in a separate package to ensure it was not permanently deleted. However, the pipeline was still available in the web interface. Any re-pushing of the package from the CLI would cause the pipeline to exist in two packages, with the web interface serving the wrong version.

Views

• Resolved formatting issues that would occur when a user with no write access was on a Views tab.

• On the Views API form, a parameter of type string was displayed incorrectly as list. This has been resolved.

Readers

• An issue where the user could not aggregate data as part of the DB Reader in the web interface, and in certain cases when using the API, has been fixed. This also includes that fill is only sent if explicitly specified.

• Using a Parquet reader with a glob pattern would error when trying to find or read a file. This has been resolved.

• Fixed an edge-case issue where a length error would occur Failed to initialize worker, worker name=[sample name] error=length when using glob patterns with file names.

• Users will no longer see a warning icon in the web interface when deploying pipelines that involve readers and writers without intra-day write-down support. Instead, an informational message will be logged. This is to be expected in this scenario and there is no action required from the user.

• The discrepancy in how AWS and Azure Cloud readers retained an extra newline character when reading in text mode without chunking, while the Google Cloud reader correctly did not, has now been resolved.

• When establishing a secret used for GCP authentication for a Parquet reader, a pipeline would fail with a Worker failed to start: Unable to locate credentials secret error.

File Watchers

• Passing in a dict for the watch parameter was not working correctly. Furthermore, use of the additional options the web interface provides when Use Watching tick box meant a dict was used, and therefore also failed. This has been resolved.

Pipelines

• In the web interface, using the "Copy pipeline" button on the Database Details page would result in the pipeline being copied, but with a default graph pipeline instead of the actual pipeline code. This has been resolved.

Third-party Dependencies

kdb Insights Enterprise 1.12.0 standalone install supports the following versions of third-party dependencies:

• cert-manager - 1.16.1
• nginx-ingress - 4.11.3
• rook-ceph - 1.15.5
• Kubernetes - 1.29

These versions are used in the 1.12.0 release of the standalone infrastructure installation scripts.

Please consult these pages for important information on supported versions:

• OpenShift
• OpenSource K8s
• EKS, AKS, and GCP

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type	Nexus location	Downloads Portal location
Enterprise	insights-1.12.0.tgz	insights-1.12.0.tgz
Operator	kxi-operator-1.12.0.tgz	kxi-operator-1.12.0.tgz
CLI	kxicli-1.12.0-py3-none-any.whl	kxicli-1.12.0-py3-none-any.whl
RT C interface	kxi-c-sdk 1.12.0	kxi-c-sdk 1.12.0
RT Python interface	kxi-rtpy-1.12.0	kxi-rtpy-1.12.0
RT Java interface	kxi-java-sdk 1.12.0	kxi-java-sdk 1.12.0
Infrastructure	kxi-terraform-1.12.0.tgz	kxi-terraform-1.12.0.tgz
kxi-management-service	kxi-management-service-0.2.4.tgz	kxi-management-service-0.2.4.tgz
insights-on-k8s	insights-on-k8s-0.2.3.tgz	insights-on-k8s-0.2.3.tgz

Upgrading to kdb Insights Enterprise 1.12

Before upgrading to 1.12 it is necessary to upgrade to the 1.12 CLI.

If upgrading to 1.11 from a prior version before to upgrading to 1.12, you would need to upgrade your CLI and ensure an encryption secret has been created first. See below for details.

You can create an encryption secret by running the kxi install create-encryption-key command as follows:

kxi install create-encryption-key

By default a secret called kxi-encryption-secret is created. If you wish to use a different name for the secret you can either:

• Specify the secret name when running the command and update your installation configuration file to use it:
  1. Run the following, replacing dare-secret-value with your chosen secret name:

     kxi install create-encryption-key --database-encryption-secret dare-secret-value
     

  2. Merge the text below into your existing installation configuration file, replacing dare-secret-value with your chosen secret name:

     global:
     encryption:
         auth:
         existingSecret: dare-secret-value
     

• Use kxi install setup and merge the output with your existing configuration file:
  1. Run kxi install setup which will automatically generate an encryption secret if one doesn't already exist, prompting you for a secret name.
  2. Merge the configuration output file from this command with your existing configuration file by referring to applying configuration changes.

With either method of generating an encryption secret, if a secret already exists with the name specified, it is not overwritten. To replace a secret, you must delete the existing secret and then a create a new one.

Read ecryption secret for more information on secrets and how to administer them.

Additional Upgrade Notes

Draft Databases & Pipelines

• As of 1.11, draft databases and pipelines that have never been saved will not be available for use. If you have draft databases or pipelines stored in your browser, they will need to be deleted. Alternatively, save your draft databases and pipelines prior to upgrade. Please be aware of this if you are upgrading to 1.11 before moving to 1.12.

Known Issues

Expand to see the list of known issues here

Entitlements

• If a user is a member of multiple user groups, their access level is based solely on the first group processed in the list of groups when checking entitlements.

Cleaning up entities

• WARNING: When tearing down a package in the web interface, the option to Clean up entities after teardown WILL DELETE ALL DATA, despite the note in the pop-up window stating the opposite: Data that has already been written down will NOT be affected. To avoid data deletion, DO NOT select the Clean up entities option. If a partial clean-up is required, use the relevant kubectl CLI commands instead.

PostGreSQL & Keycloak Resiliency

• A transient issue can occur in which Keycloak is unable to re-establish a connection to PostgreSQL after it recovers from an outage. This can result in getData being unable to serve queries. To work around this, restart PostgreSQL.

Upgrades & Rollbacks

• Users are prompted to tear down assemblies upon rollback. The rollback should exit if the user responds ‘n' to this prompt. Instead, the rollback continues.

Deprecations

• The kxi package commands for remote operations has been deprecated as of the 1.12 release. kxi pm offers full functional equivalency.

  • pull
  • push
  • remote-list
  • remote-remove
  • remote-convert
  • remote-config
  • deploy
  • teardown

• The RT Bridge has been deprecated as of the 1.12 release. The rt.qpk now offers full functional equivalency to the RT bridge so it is recommended to migrate to it prior to RT Bridge being removed.

Summary

We hope you find some useful features that optimize your kdb Insights Enterprise experience. Try them out and email our Support Team if you need any help.

We look forward to bring you even bigger features in kdb Insights Enterprise 1.13 coming soon!