kdb Insights Enterprise - Release Notes

This page details the release notes for the latest series of kdb Insights Enterprise releases. Key themes include improved access controls, usability and faster ingestion of real-time data into the database. To find the release notes for an older release, refer to the previous releases page.

1.9.0

Release Date

2024-03-26

New Features

• Developer Experience Improvements for Packaging

  kdb Insights Enterprise packaging now allows clients to manage database and pipeline configuration as well as custom code within a single package. This greatly simplifies developer and operational workflows, as users can maintain and administer databases and pipelines within a single package and with specific configuration for each pipeline. Packaging has also been integrated with the UI so that any databases and pipeline created using the CLI using the kxi package command can be viewed and maintained by the UI and vice versa. Refer to Packaging for more details.

• SQL Enhancements

  SQL in kdb Insights Enterprise now includes support for ORDER BY and LIMIT clauses, in addition to other functions. Changes to how SQL statements are parsed also significantly improves SQL query performance in this release. Query performance, on average, is up to 3x faster than before. Refer to SQL 2 for more information on the new SQL implementation and its current Known Limitations.

• Streaming to Views

  In the kdb Insights Enterprise UI, Views can now subscribe to real-time data from a pipeline, rather than relying on intermittent polling. This gives you the ability to visualize and analyze your data immediately. Refer to the streaming guided tour for details on configuring pipelines and views for this feature.

  Streaming to KX Views is a BETA feature in kdb Insights Enterprise 1.9.0.

  · Refer here to the Standard Terms related to Beta features.

  · We invite you to use this Beta feature and to provide feedback using the portal.

  · Refer here to the known limitations of this Beta feature.

• Snapshots & Majority Archival for Reliable Transport

  Reliable Transport now supports taking snapshots allowing for majority archival of logs files. Previously, all nodes needed to be healthy before RT logs could be archived.

• Configurable Connections for Reliable Transport

  Reliable Transport now supports subscribers and publishers setting customized parameters when they register with their respective sequencer cluster. This gives you greater configurability in how you deploy and orchestrate components of RT. See rt.qpk for more details.

Improvements

• System Resources Configuration

  Defining system resources has been streamlined, enabling you to set all aspects of resource allocation from a single configuration page in the kdb Insights Enterprise UI. See Adjust Resources for more information on this new tab.

• Batch Delete

  Batch delete now allows you to delete from all tiers of your HDB. Previously, this was only possible for the tier with the most recent HDB data. See Database Delete for details on how to perform a delete.

• Python Cloud Readers

  Python cloud readers now support polling for new matches to a glob pattern with the watch parameter. See Readers for details.

• Upgrade to Insights Core 4.1.1

  kdb Insights Enterprise images have been upgraded to run Insights Core 4.1.1, which runs kdb+ 4.1 2024.03.12.

• Upgrade to RockyLinux 9

  kdb Insights Enterprise images have been upgraded to run on Rocky Linux version 9 as their base operating system. This addresses security vulnerabilities present in version 8.

• Stream Processor Troubleshooting Guide

  There is now a troubleshooting guide available to help users better understand how to debug their pipelines running in kdb Insights Enterprise.

Fixes

Expand to see the full detail of resolved issues here

Database

• If EOD failed while in the middle of an HDB transaction, the aborted transaction was not properly cleaned up after the failure and could result in data inconsistency after reconnection.

• The EOD process was not performing a garbage collection after a batch ingest.

• Data was unqueryable after a batch ingest if no additional streaming data was loaded after the initial batch ingest.

Reliable Transport

• In deployments with a one-node RT configuration, RT has been updated to roll its log file after a restart. This prevents data flow from being paused. The RT log file will roll from log.x.y to log.x.y+1.

UI

• The Subscriber node was limited to only specifying one keyed column. This has been resolved to allow one or more keyed columns, ensuring that streaming supports a record being streamed per unique combination of all specified keyed columns and to allow filtering on any one of the keyed columns.

• When configuring a Schema node, the regular expression pattern for table validation has been updated to allow single character column names.

• For Function nodes, users can now customize their function parameters to accept the metadata and operator, in addition to data, in the Pipeline UI. See here for details.

• The Error Mode in the Pipeline Settings section of the Pipeline editor has been removed as it is no longer required, due to a stack trace now being logged when user code errors.

• The logging level within the Runtime Pipeline Settings section of the Pipeline editor was not correctly displaying or setting the log level of the SP worker. This has been resolved.

• There was no way to specify \t as the delimiter in a CSV decoder in the UI. This has been corrected in the CSV Decoder node to allow for any delimiter to be used.

• After a pipeline has been deployed, when the status is either “Running” or “Finished”, a user is able to delete the pipeline. When the pipeline is deleted, the Overview page was still displaying it under “Running Pipelines”. This has been resolved to not display deleted pipelines.

• Disk sizes for IDB and HDB tiers can now be set to any value greater than 0.

• Users without the permission to create a pipeline should not be able to use the Import Data button on Database Page. This has been resolved.

• Deleting running databases is now allowed.

Views

• Notifications in Views were not reporting correctly. This has been resolved.

• Under certain circumstances when you had a particular data source selected in the Data Source window and clicked New the existing values remained visible, however they were all ignored. This has been resolved.

• When using the getData API with agg set to sym;distinct;sym, the UI was ignoring this request and not sending any aggregation. This has been resolved.

• Switching between tabs caused the user-selected date to switch back to the default setting. This caused the View to constantly reload. This has been resolved.

Stream Processor

• When using Python, 'sp.decode.csv' did not support reading columns as strings. This has been updated to enhance flexibility when reading table columns. It now accepts an empty string "" in lieu of a schema to read in all table columns as strings.

• Sidecar containers no longer log at a TRACE level by default and now adhere to the log level that is set by the user.

• The SP controller now adheres to the log level set by the user.

• The CSV decoder did not support decoding csvs which had line-returns (i.e. \n’s) embedded in strings. This functionality can now be turned on with a new optional parameter called Strings Contain Newlines in the UI and newlines in the q and Python APIs.

• When running a large import from a static source (file, Azure, AWS, GCP), there was no indication of progress in the logs. The SP logs now displays the file offset periodically while processing to indicate on-going progress of the import.

• The SP’s status APIs will now return TEARING_DOWN rather than NOTEXIST when a pipeline has PVCs that are still in a terminating state.

• Under certain circumstances when you added an additional stream to your database it could become orphaned after you renamed the database. This has been resolved.

• Under certain circumstances when creating a pipeline using the Import feature the Parse Strings column was missing from the Apply Schema page, causing the pipeline to fail when deployed. This has been resolved.

• The running pipelines section displays a count of warnings and errors that have occurred in the last 10 minutes. Not all of these warnings were being displayed in the Diagnostics page. This has been resolved.

CLI

• When trying to enroll a client with a topic, the client was enrolled but the specified topic was not being set. This has been resolved.

• When a user tried to authenticate but did not have a local browser installed on their machine, the redirect URL was localhost instead of the value specified. This has been resolved such that users with a local browser installed can still successfully authenticate.

• Installation with a kc.lic license using the --license-filepath command-line parameter was generating an incorrect values.yaml file. This has been resolved so the license values correspond to the license name.

• Specifying a different hostname as part of a kxi command is ignored after authentication, as you can only be authenticated to one kdb Insights Enterprise host at a time. For example, if you authenticate to HOSTNAME1 by calling kxi --hostname HOSTNAME1 auth login you are free to make calls against this host. If you attempt to specify a different hostname in subsequent calls kxi --hostname HOSTNAME2, the --hostname parameter will be ignored and the command executed against HOSTNAME1. You must re-authenticate against HOSTNAME2 before running commands against it.

• The function .kxi.udfs.list.all[] was failing with a length error. This has been resolved.

• Global environment variables set within your values.yaml will now be set by all database and assembly deployments.

  global:
  env:
      env: var
  

Query Environments

• Query environments with entitlements enabled required the creation of separate entitlement for the -qe assembly in order for queries to work. This has been resolved.

Upgrades

• Fixed an issue where it was possible for pods to be unscheduled after an upgrade or rollback due to a volume node affinity conflict. Please note that this is still possible when rolling back from 1.9 to earlier versions. Please check the release notes of the target rollback version to address.

Security

• Various CVEs have been remediated as part of this release.

Third-party Dependencies

kdb Insights Enterprise 1.9.0 standalone install supports the following versions of third-party dependencies:

• cert-manager - 1.13.3
• nginx-ingress - 4.9.0
• rook-ceph - 1.13.1
• Kubernetes - 1.28

These versions are used in the 1.9.0 release of the standalone infrastructure installation scripts.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type	Nexus location	Downloads Portal location
Enterprise	insights-1.9.0.tgz	insights-1.9.0.tgz
Operator	kxi-operator-1.9.0.tgz	kxi-operator-1.9.0.tgz
CLI	kxicli-1.9.0-py3-none-any.whl	kxicli-1.9.0-py3-none-any.whl
RT C interface	kxi-c-sdk 1.9.0	kxi-c-sdk 1.9.0
RT Python interface	kxi-rtpy-1.9.0	kxi-rtpy-1.9.0
RT Java interface	kxi-java-sdk 1.9.0	kxi-java-sdk 1.9.0
RT Bridge	rt-bridge 1.7.0	rt-bridge 1.7.0
Infrastructure	kxi-terraform-1.9.0.tgz	kxi-terraform-1.9.0.tgz

Upgrade Notes

• Keycloak and Postgres Values Changes

  The values file settings for the Keycloak and Postgres components have changed for 1.9. The panels below describe the modifications you need to make to this file for all standard or shared Keycloak deployments before upgrading to 1.9.

  Merging install values

  The values below are a partial extract of your deployment values. These should be merged with your existing values file.

  For more information, refer to Applying configuration changes.

  Standard DeploymentShared Keycloak

  A standard deployment deploys Keycloak as part of the application. If you're using this configuration, you should ensure your values file includes the details shown below.

  The names kxi-keycloak and kxi-postgresql are the default values, update these as appropriate.

  global:
    keycloak:
      auth:
        existingSecret: kxi-keycloak
    postgresql:
      auth:
        existingSecret: kxi-postgresql
    ..
  
  keycloak:
    auth:
      existingSecret: kxi-keycloak
    postgresql:
      auth:
        existingSecret: kxi-postgresql
  ..
  

  Keycloak values

  Please note there are two sets of keycloak and postgres objects; global.keycloak, global.postgresql, keycloak, keycloak.postgresql.

  Both sets of these are required as shown above.

  If deploying using the shared Keycloak mode, configure the values file as described below.

  The KEYCLOAK_RELEASE_NAME and KEYCLOAK_NAMESPACE variables should be updated to match those of your Keycloak deployment. The shared Keycloak docs explain each of these values.

  global:
    keycloak:
      authURL: http://${KEYCLOAK_RELEASE_NAME}.${KEYCLOAK_NAMESPACE}.svc.cluster.local/auth/
      auth:
        existingSecret: kxi-keycloak
    postgresql:
      auth:
        existingSecret: kxi-postgresql
    ..
  

• If a user has an existing kxi-license secret created from a kc.lic or k4.lic, kxi install setup will re-use this secret, but will not set the appropriate values in the resulting values.yaml file. See available license types Setup - kdb products.

• The default image repository used for new 1.9 installations will be the KX Downloads Portal. The credentials to access this are stored in a docker config json secret named kxi-image-pull-secret. If you are upgrading from an earlier version that used the KX Nexus image repository with an exiting secret kxi-nexus-pull-secret, this existing repository and secret will be used.

• After executing an upgrade, if an RT pod does not appear as ‘Ready’, the pod should be deleted/killed so that it is restarted.

• During an upgrade, the RT pods attempt to delete a Kubernetes configmap. Since there were 3 RT pods, only one of the pods will be successful in deleting the config map. This means that the other pods will receive ERRORs back indicating that their delete attempt failed. These Failed to delete snapshot: tmp errors are benign and will be set to WARN messages in future.

• Query Labels Style

  KXI_ALLOW_OLD_LABEL_STYLE now defaults to "false". It must be explicitly set to "true" in the RC to allow the "old label style". For example, in a system with labels region, and class:

  • Labels in queries (such as .kxi.getData) must be defined in the labels parameter.

    • old style: --data '{"table": "trade", "region": "usa", "class": "equities"}'
    • new style (new default): --data {"table": "trade", "labels": {"region": "usa", "class": "equities"} }'

  • Labels in SQL queries must be prefixed with label_.

    • old style: SELECT * FROM trade WHERE region='usa', class='equities'
    • new style (new default): SELECT * FROM trade WHERE label_region='usa', label_class='equities'

Rollback

• If performing a rollback from 1.9.0 to any earlier version, assemblies must be running prior to initiating the rollback to ensure no data is lost. See Upgrading for how to perform upgrades/rollbacks and the available commands to use. Please see the following specific instructions per version:

  Rollback from 1.9 to a version less than 1.7 is not possible.

  For rollback from 1.9 to a prior version:

  1. Confirm all assemblies are running before initiating rollback.
  2. Execute the rollback operation. Note that the kxi-rt-rollback and kxi-sp-rollback charts are run automatically as part of the rollback operation as of 1.9.0.

  For rollback from 1.8, the following instructions should be followed:

  1. Teardown your assemblies.
  2. Run the kxi-sp-rollback chart. If the target version for rollback is less than 1.7, also run the kxi-db-rollback chart.
  3. Rollback kdb Insights Enterprise per the rollback instructions referenced above.
  4. Redeploy assemblies.

Known Issues

Expand to see the list of known issues here

Upgrades & Rollbacks

• Users are prompted to tear down assemblies upon rollback. The rollback should exit if the user responds ‘n' to this prompt. Instead, the rollback continues.

• Occasionally, rollbacks may fail with an error when running kxi install rollback. This is due to timeouts with the Helm rollback, resulting in errors such as the following:

      timed out waiting for the condition
      context deadline exceeded
  

  To resolve this issue, re-run the rollback.

Pipelines

• It is currently possible to rename a pipeline in the UI to a pipeline name that is already in use. Until this is resolved, ensure your pipeline names are unique.

• When deploying a pipeline that writes to a table that does not exist, no error will be thrown.

Packaging

• kxi.package.load_file does not work properly for Python files. For a Python module, use import instead.

Query Tab

• There is presently no way to select a subset of the available columns during the getdata phase in the Query tab of the UI. Conversely, the getData API allows this in the agg parameter where you can pass in the list of columns you want with no aggregation function.

UI

• Changing the system log level in the UI does not get actioned.

• On Mozilla Firefox, when clicking Deploy in a Database screen, the scroll wheels do not render properly in the pop-up.

Views

• When displaying streaming data in Views and using the Map component, the map may take some time before the streaming data is displayed. A page reload may be required.

• When working with a View then temporarily navigating away to a different tab, any changes made will be saved despite the user not requesting this. If multiple users are editing the same view, any time one switches away from the tab (even if they haven’t made any changes), the current version of their view will be saved, which may not include changes made other users. To resolve this, use the Undo action to return the last desired saved state.

• A popup may be shown indicating there may be unsaved changes when no changes have been made.

Scratchpads

• Certain Python expressions cause internal errors in Python scratchpads:

  lambda x: x * 2
  memoryview(bytearray("myTest", encoding='utf-8'))
  {1,2,3}
  

Reliable Transport

• RT clients output certain logs to a folder determined by theKXI_C_SDK_APP_LOG_PATH environment variable ($KXI_C_SDK_APP_LOG_PATH/kxi_c_sdk_logs/rt_helper.VMC7AW/rt_helper.log). This variable uses a default location of /tmp. When there are recurring errors, these logs can fill the /tmp directory which is only 5MB in size. To overcome this issue, you can change this environment variable. See Environment variables for steps on how to change this.

Deprecation

• The CLI kxi assembly command is being deprecated in 1.9.0 in favor of kxi package and may be removed from future releases. See kdb Insights CLI Reference for more information on the kxi assembly command.

• prtnEndCB and reloadCB callback functions are being deprecated in 1.9.0 and may be removed from future releases. They have been replaced by lifecycle.partition.end and lifecycle.reload.post respectively. Their usage can be seen below.

.custom.da.prtnEndCB:{[event;data]
    log.info("lifecycle.partition.end invoked.")
    }
.custom.da.reloadCB:{[event;data]
    log.info("lifecycle.reload.post invoked.")
    }

.ev.add[`lifecycle.partition.end ;`.custom.da.prtnEndCB];
.ev.add[`lifecycle.reload.post ;`.custom.da.reloadCB];

• RT maxLogSize is no longer used in configuration and will be ignored from older configurations. RT will now utilize all disk available to it.