Skip to content

kdb Insights Enterprise - Release Notes

This page details the release notes for the latest series of kdb Insights Enterprise releases. To find the release notes for an older release, refer to the previous releases page.

1.11.1

Release Date 2024-11-04

Improvements

Parquet Reader

  • A local file watcher has been added for the Parquet reader. See q and Python readers documentation for details.

  • The Parquet reader now supports specifying the file path as a symbol.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Resolved an issue where a query for an empty date could fail when late data is turned on.

  • Previously, a failed end-of-day writedown (EOD) was not retried by the Storage Manager, causing a misaligned error on the next EOD.

Views

  • When using user-defined analytics in Views, the parameter of type string (10h) was displayed as a list. This has been resolved to display properly as a string.

  • When binding a viewstate to a string parameter, the binding was incorrectly removed after saving the data source.

Pipelines

  • The health status of a pipeline is now displayed on the Running Pipelines section of the homepage, in addition to the Pipelines page. Previously, these were displaying different types of information about pipeline status which was not clear.

Packaging

  • Resolved an issue where filtering by name, id, etc. was not working correctly when listing pipelines or databases in the CLI (kxi package remote-list). This has been resolved.

  • Resolved an issue where a package containing Patch objects could not be deployable.

  • Fixed a bug with kxi package packit with --package-name and/or --package-version. The original package will remain unmodified and the artifact will contain the source code.

  • Fixed unresponsive package manager server (and UI). The connection between packagemanager and Kubernetes now has a timeout, meaning that any stale connections will be dropped after 15 seconds, limiting any unresponsiveness to this timeframe.

Scratchpad

  • The scratchpad manager was not setting the no proxy environment variable during install, as well as when any new pod was created as part of user login. This was manifesting as auth errors in the UI when using the scratchpad.

Entitlements

  • On the Pipeline screen, if you do not have the entitlements to both save and deploy a pipeline the "Save and Deploy" button has been separated into two distinct buttons. This addresses an issue in which users without write access permissions were unable to deploy a pipeline because they did not have permissions to save it.

Permissions

  • Users with the Viewer composite role were unable to use qsql without adding the insights.query.custom permission. This caused views to appear broken after upgrading to 1.11.

Reliable Transport

  • In the getLatestPosition API in the rt.qpk, the dictionary key peers was not functional in 1.11.0. This has been resolved.

Readers

  • Fixed an edge-case issue where an error occurred when using glob patterns with file names.

  • When establishing a secret used for GCP authentication for a Parquet reader, a pipeline would fail with a Worker failed to start: Unable to locate credentials secret error.

  • Using a Parquet reader with a glob pattern would error when trying to find or read a file. This has been resolved.

Security

  • Various CVEs were remediated as part of this release.

Third-party Dependencies

kdb Insights Enterprise 1.11.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.11.0 release of the standalone infrastructure installation scripts.

Artifacts

KX Nexus

The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users.

type Nexus location Downloads Portal location
Enterprise insights-1.11.1.tgz insights-1.11.1.tgz
Operator kxi-operator-1.11.0.tgz kxi-operator-1.11.1.tgz
CLI kxicli-1.11.1-py3-none-any.whl kxicli-1.11.1-py3-none-any.whl
RT C interface kxi-c-sdk 1.11.0 kxi-c-sdk 1.11.0
RT Python interface kxi-rtpy-1.11.0 kxi-rtpy-1.11.0
RT Java interface kxi-java-sdk 1.11.0 kxi-java-sdk 1.11.0
Infrastructure kxi-terraform-1.11.0.tgz kxi-terraform-1.11.0.tgz
kxi-management-service kxi-management-service-0.2.3.tgz kxi-management-service-0.2.3.tgz
insights-on-k8s insights-on-k8s-0.1.6.tgz insights-on-k8s-0.1.6.tgz

1.11.0

Release Date 2024-09-20

kdb Insights Enterprise 1.11 brings increased security measures with data at rest encryption, continued improvements to the user experience with better visibility into service health, enhancements to stability and performance and so much more! See below for the full details.

New Features

  1. Data At Rest Encryption (DARE)
  2. Scratchpad optimizations
  3. Service health observability
  4. HTTP proxy support
  5. Reliable Transport's (RT) client replica size discovery
  6. Improvements to user-defined analytics
  7. PDF generation for Views
  8. LIMIT support in getData
  9. Live rolling View States

1. Data At Rest Encryption (DARE)

Encrypt all tables in your database using kdb+'s native DARE capability. When enabled, all on-disk data is encrypted using AES256CBC. DARE enables you to comply with your data security requirements. Read the encryption at rest section for details.

2. Scratchpad optimizations

Scratchpad optimizations provide a seamless scratchpad experience, ensuring responsiveness and minimizes the need for restarts. It addresses potential connectivity issues, CPU saturation, and memory constraints, while maintaining performance for uninterrupted service. Additionally, it provides scalable capacity to support user growth, significantly enhancing reliability and overall user satisfaction. Read the query docs for details on scratchpads. Note that previously, the default allocation for scratchpad resourcing was 8GB shared across all users - this has been updated to a default of 500MB per user. As a result, it is recommended you assess your workloads and user counts and then set your Scratchpad resourcing appropriately.

3. Service health observability

Enhancements to observability provide clearer visibility into the health of the underlying services, particularly where data flows through the system. You can now quickly identify and trace problems in your system without interrogating multiple service logs. This improvement speeds up root cause analysis and resolution, significantly reducing troubleshooting time. Read the health section for details.

4. HTTP proxy support

Configure seamless integration with customer proxies. Global environment variables (http_proxy, https_proxy, and no_proxy) are propagated to sub-components and third-party services, ensuring correct traffic routing and reliable pipeline deployment. Note this feature is only supported for Azure-based deployments, as only Azure Kubernetes Service supports kubelet services behind a proxy. Refer to deploying behind a proxy server for details.

5. Reliable Transport's (RT) Client Replica Size Discovery

Enhancements to RT simplify client connection and configuration setup. This includes automatic discovery of the cluster size, eliminating the need for clients to specify sizing manually and allowing for seamless resizing of the RT cluster. This change enhances the user experience by reducing the configuration needed and streamlining setup across all interfaces (C, q, Python, and Java). Explore the enhancements in Getting started.

6. Improvements to User Defined Analytics (UDAs)

We have enhanced the documentation and consolidated User Defined Analytics (UDAs) development into a single, unified API. This update simplifies the process for developers by eliminating the need for detailed internal system knowledge and streamlines API functionality across various components. This release also introduces the ability for a UDA to call another UDA. Read the UDA introduction for an overview of UDAs and details on how to use the new functions. In addition, please read the Deprecations section below, which outlines name changes to important functions.

7. PDF generation for Views

A PDF can now be created from your View(s) in kdb Insights Enterprise. This supports single and multiple tabs. Additionally, the Share Link View now supports dark mode and aligns with the dashboard design for a unified appearance. To learn how to use these new features, read the new docs.

8. limit support in getData

The getData API now supports a limit clause, allowing you to retrieve a subset of data (for example, the top 100 rows). This enhancement increases the reliability of queries on large datasets and offers more efficient data handling. It improves the user experience when running large queries, reducing the risk of the query failing. Read limit for details on how to use limit in getData. Refer to row limit for guidance on setting limits in the Query builder. You can also take advantage of this when using Views.

9. Live Rolling View States

View States have been enhanced to support rolling syntax to execute at times other than load time. Rolling syntax applies to temporal data and allows you to provide relative timestamps to be used in data source queries. Rolling view state can now be configured to run at dashboard load and at the data source’s polling interval. View the docs for details.

New Beta Features

Entitlements for Packages

Secure package creation and deployment, provides entitlements for user-created packages via the CLI. This ensures secure collaboration by protecting deployments and analytics from unauthorized access or changes. Read package entitlements for details. For reference, a full overview of packages can be found here and an overview of entitlements can be found here.

Improvements

Performance improvements for Storage Manager operations

  • Optimizations in the Storage Manager have been made to improve the efficiency of schema conversions, including the introduction of multi-threading. This significantly reductes the time taken to perform a schema change operation. Read multi-threading for more details.

  • Reduction in the RAM required for re-enumeration when carrying out a batch ingest.

  • Adding/removing an object storage tier to existing databases is now supported. See here for details.

Technology Migration for 3D Charts and Dashboards

jQuery and jQueryUI have been removed from 3D charts, and various Dashboard components have been migrated to newer technologies. This update addresses potential vulnerabilities related to cross-site scripting (XSS) and DOM manipulation, ensuring a more secure environment.

SQL

We have introduced support in SQL2 for the following:

  • a new keyword OFFSET

  • multi-directional ORDER BY. An example is seen below.

    select * from trade order by sym asc, price desc

getData API

The following improvements have been introduced in 1.11:

  • support for descending and mixed sorting in getData has been added.
  • getData now supports OFFSET as part of its limit implementation. Read getData limits for details.
  • getData now validates that functions are called with the correct number of parameters.
  • the ability to bind view state variables easily into filter statements in getData. Refer to using-view-states for details.

File Watchers

  • The local file reader now supports file watching. See the documentation for q and Python for details.

Pipelines

  • Parallel operator nodes have been added to the UI node library. With this, users can add multiple functions to run in parallel and define merge functions to merge their outputs.

  • The Pipeline editor now supports using the Ctrl+D/⌘D hotkey for executing code (in both Windows and Mac).

Scratchpads

  • The Stream Reader can now be used in a scratchpad by simply calling the API as documented. There is no longer the need to set environment variables KXI_SP_CHECKPOINT_DIR and RT_TOPIC_PREFIX.

  • The Query window now supports specifying Ascending/Descending option for Sort filters.

  • Scratchpad now supports using the Ctrl+D/⌘D hotkeys for executing code (in both Windows and Mac).

Accessibility

  • Improvements have been made to the screen reader and keyboard accessibility:
    • On the Overview pages, the table listing pages now have a hidden element for screen readers that reads: “This table has no data" when no data is available for a table
    • Elements are read in the order they appear on screen (Asc or Desc)
    • The 'i' information icon on the Database setting screen is now reachable with the keyboard using the tab key
    • In Database settings → Tier settings, the clock icon now displays a gray background when it is selected with the keyboard

CLI

  • Improvements have been made to how the kxicli handles failures in installation or upgrade in the following ways:
    • failure reasons are now printed after each retry
    • default number of retries has been updated to 2
    • timeouts are now exponentially increasing e.g. to 5m on the first try, increasing to 10m on the second try
    • wait times are also exponentially increasing between retries
    • provide additional options:
      • set the number of retries using the --retry-limit command
      • set the initial wait between retries using the --retry-wait command

Packages

  • Packaging now allows the pushing of later versions of deployed packages, even while old versions are running. Previously, all pushes were blocked when a package was deployed.

  • Error messages are now clearer when an error is encountered during package load, showing the offending filename, line number and line contents if there is a syntax error in the schema definition.

  • kxi package now provides a straightforward way to view and set configuration parameters: 

    kxi package config get      Get local package manager configuration.
kxi package config set      Set local package manager configuration values.
    For more information, run the command kxi package config --help.

  • The command kxi package teardown now takes a list. This allows you to provide more than one package to be torn down in a single command, and also supports use of regex strings to specify the targeted packages. See here for details.

  • The details and readability of the kxi package field info command has been improved with the following:

    • Better descriptions of each type
    • Color highlighting
    • Helpful messages
    • Improved error messages when package definitions are incorrect
  • IDE Package files now include autocomplete and validation when modifying packages.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Fixed an issue where getData queries failed with a type error if the time range of the request landed completely between two IDB partitions.

  • Fixed an issue where getData was not correctly enlisting the target symbol when using <> like it does when using =.

  • getData now validates that functions are called with the correct number of parameters.

  • getData filters of the form ["like", "col", x] where col is a string/symbol column and x is a string, symbol, or char, now work without throwing a type error.

  • Fixed an issue where getData REST queries would produce inf instead of a null.

  • Batch ingest direct write pipelines now correctly honor compression and encryption settings of an HDB tier.

  • In the Advanced settings page on the Database configuration screen, it was possible to set the retention time for the HDB to a value less than one day using hours/minutes/seconds. This is not supported, and has been corrected so values less than one day are rounded up to a day upon loading.

  • Previously, there was no way for a user to specify the inventory path for the object storage tier of their HDB from the UI. This is now possible to configure via the Tier Settings in the UI. Providing an inventory path allows faster subsequent reloads for the SM and DAPs.

Pipelines

  • When configuring a pipeline in the UI, setting additional workerThreads did not correctly apply the specified configuration. This has been resolved.

  • Custom component fields were not being disabled correctly on deploy. This affected the Use Watching functionality, and the Dictionary, Apply Schema, and CSV Decoder Delimiter fields on the Pipelines screen. This has been resolved.

  • Fixed issues with undo/redo options in the pipeline editor.

  • Pipelines were erroneously showing that they had unsaved changed in the tab bar and entity tree on the Pipelines page, even when no changes had been made. This has been resolved.

  • Previously, pipelines were allowed to be removed from a running databases. This has been resolved to block this action.

Readers

  • Resolved an issue where the Database reader was throwing type errors when filters were applied. These queries now function correctly, including in the Scratchpad UI.

UI

  • Previously, different entity types (pipelines, views, queries, etc) could not have duplicate names. This has been corrected such that entities of different types can have the same name, but entities of the same type cannot.

  • A differentiator has been added to the tab menu to distinguish between a tab focused on a pipeline and a tab focused on the diagnostics of either a database or pipeline. This reduces confusion of multiple tabs with the same name.

Packages

  • The packaging CLI will now print a warning when its version doesn’t align with the package-manager server.

Scratchpads

  • If using Mozilla Firefox, the scratchpad UI was accidentally switching tabs when selecting text. This has been resolved.

CLI

  • When using the debug option in the CLI, kxi --debug install was not being propagated throughout the install process. This has been resolved.

  • Users were not being notified of issues when removing an enrolled client. This has been resolved.

Security

  • Various CVEs have been remediated as part of this release.

Third-party Dependencies

kdb Insights Enterprise 1.11.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.11.0 release of the standalone infrastructure installation scripts.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type Nexus location Downloads Portal location
Enterprise insights-1.11.0.tgz insights-1.11.0.tgz
Operator kxi-operator-1.11.0.tgz kxi-operator-1.11.0.tgz
CLI kxicli-1.11.0-py3-none-any.whl kxicli-1.11.0-py3-none-any.whl
RT C interface kxi-c-sdk 1.11.0 kxi-c-sdk 1.11.0
RT Python interface kxi-rtpy-1.11.0 kxi-rtpy-1.11.0
RT Java interface kxi-java-sdk 1.11.0 kxi-java-sdk 1.11.0
Infrastructure kxi-terraform-1.11.0.tgz kxi-terraform-1.11.0.tgz
kxi-management-service kxi-management-service-0.2.3.tgz kxi-management-service-0.2.3.tgz
insights-on-k8s insights-on-k8s-0.1.6.tgz insights-on-k8s-0.1.6.tgz

Upgrading to kdb Insights Enterprise 1.11

Before upgrading to 1.11 it is necessary to upgrade to the 1.11 CLI and ensure an encryption secret has been created.

You can create an encryption secret by running the kxi install create-encryption-key command as follows: 

kxi install create-encryption-key

By default a secret called kxi-encryption-secret is created. If you wish to use a different name for the secret you can either:

  • Specify the secret name when running the command and update your installation configuration file to use it:
    1. Run the following, replacing dare-secret-value with your chosen secret name: 
      kxi install create-encryption-key --database-encryption-secret dare-secret-value
    2. Merge the text below into your existing installation configuration file, replacing dare-secret-value with your chosen secret name: 
      global:
encryption:
    auth:
    existingSecret: dare-secret-value
  • Use kxi install setup and merge the output with your existing configuration file:
    1. Run kxi install setup which will automatically generate an encryption secret if one doesn't already exist, prompting you for a secret name.
    2. Merge the configuration output file from this command with your existing configuration file by referring to applying configuration changes.

With either method of generating an encryption secret, if a secret already exists with the name specified, it is not overwritten. If you want to replace one, you must delete the existing secret and then a create a new one.

See here for more information on secrets and how to administer them.

Additional Upgrade Notes

Draft Databases & Pipelines

  • As of 1.11, draft databases and pipelines that have never been saved will not be available for use. If you have draft databases or pipelines stored in your browser, they will need to be deleted. Alternatively, save your draft databases and pipelines prior to upgrade.

Known Issues

Expand to see the list of known issues here

PostGreSQL & Keycloak Resiliency

  • A transient issue can occur in which Keycloak is unable to re-establish a connection to PostgreSQL after it recovers from an outage. This can result in getData being unable to serve queries. To work around this, restart PostgreSQL.

Pipelines

  • Pipelines that use Apply nodes from releases older than 1.9 will fail on deployment after upgrading. To work around this, after performing your upgrade, open the pipeline, click on Apply node, click the gear icon in the top corner of the node-config area, and then set the parameters manually (for most apply nodes this will be operator, metadata, data).

  • Deploying a pipeline from the UI will mark it as having unsaved changes even when none exist. Clicking Save again will make this go away.

Upgrades & Rollbacks

  • Users are prompted to tear down assemblies upon rollback. The rollback should exit if the user responds ‘n' to this prompt. Instead, the rollback continues.

Packaging

  • If using entitlements (currently in BETA), you cannot load a package in the scratchpad using Python or using a Python UDF.

  • In the UI, when you remove a pipeline from a database, packaging preserves this pipeline in a separate package to ensure it is not permanently deleted. This means the pipeline is still available in the UI. However, if you re-push the package from the CLI that still contains the pipeline, the pipeline now exists in two packages, and the latest version of the pipeline is not being served to the UI. The workaround is to remove the pipeline-only package that has been created.

New Requirements for Hostnames & kxi package

  • When you run a kxi package command, the kxi cli first authenticates against kdb Insights Enterprise. In 1.9, running a kxi package command would look up the cached hostname that had been previously authenticated against in the ~/.insights/credentials file. In 1.10.0, the hostname previously authenticated against is no longer used. Instead, you must set the hostname in your ~/.insights/cli-config.

Deprecations

  • As a result of the improvements to how user-defined analytics work, the following functions are being deprecated, but are still be available to use:

    • .kxi.selectTable has gone from a 6-parameter function to a single function that takes a single dictionary argument.
    • .da.getTableMem replaced by .kxi.getTableBuffer
    • .da.getTableDelta replaced by .kxi.getTableOverflow
    • .da.i.dapType (variable) replaced by .kxi.getDapType function
    • .sapi.{ok,hok} replaced by .kxi.response.{ok,hok}
    • .sapi.response replaced by .kxi.response.make
    • .sapi.defer replaced by .kxi.response.callAPI
    • DAP UDAs must return a response with one of the .kxi.response.{*} functions, e.g. .kxi.response.ok
    • .da.registerAPI and .sgagg.registerAggFn replaced by .kxi.registerUDA
    • .sapi.meta{Param,Return,Description,Misc} replaced by .kxi.meta{Param,Return,Description,Misc}
    • .kxi.getMeta v2 replaced by v3.
    • KXI_SAPI_HB_{FREQ,TOL} replaced by KXI_HB_{FREQ,TOL}

  • When configuring a process (Aggregator, DAP, RC) , the environment variable KXI_SAPI_HB_FREQ set the time in milliseconds to run the heartbeat to connected processes, and KXI_SAPI_HB_TOL to set the number of heartbeat intervals a process can miss before being disconnected (default: 2). These variables have been updated to remove the deprecated reference to SAPI (see above). These configurations are now KXI_HB_FREQ and KXI_HB_TOL respectively. The old values will be supported but are deprecated, and will be sunsetted in future.

Summary

We hope you find some useful features that optimize your kdb Insights Enterprise experience. Try them out and email our Support Team if you need any help.

We look forward to bring you even bigger features in kdb Insights Enterprise 1.12 coming soon!