Skip to content

kdb Insights Microservices - Release Notes

This page details the release notes for the latest series of kdb Insights Microservices releases. Key themes include improved access controls, usability and faster ingestion of real-time data into the database. To find the release notes for an older release, refer to the previous releases page.

1.9.0

Release Date

2024-03-26

New Features

  • SQL Enhancements

    SQL in kdb Insights now includes support for ORDER BY and LIMIT clauses, in addition to other functions. Changes to how SQL statements are parsed also significantly improves SQL query performance in this release. Query performance, on average, is up to 3x faster than before. Refer to SQL 2 for more information on the new SQL implementation and its current Known Limitations.

  • Snapshots & Majority Archival for Reliable Transport

    Reliable Transport now supports taking snapshots allowing for majority archival of logs files. Previously, all nodes needed to be healthy before RT logs could be archived.

  • Configurable Connections for Reliable Transport

    Reliable Transport now supports subscribers and publishers setting customized parameters when they register with their respective sequencer cluster. This gives you greater configurability in how you deploy and orchestrate components of RT. See rt.qpk for more details.

Improvements

  • Batch Delete

    Batch delete now enables you to delete from all tiers of your HDB. Previously, this was only possible for the tier with the most recent HDB data. See Database Delete for details on how to perform a delete.

  • Python Cloud Readers

    Python cloud readers now support polling for new matches to a glob pattern with the watch parameter. See Readers for details.

  • Upgrade to Insights Core 4.1.1

    kdb Insights images have been upgraded to run Insights Core 4.1.1, which runs kdb+ 4.1 2024.03.12.

  • Upgrade to RockyLinux 9

    kdb Insights images have been upgraded to run on Rocky Linux version 9 as their base operating system. This addresses security vulnerabilities present in version 8.

Fixes

Expand to see the full detail of resolved issues here

Database

  • If EOD failed while in the middle of an HDB transaction, the aborted transaction was not properly cleaned up after the failure and could result in data inconsistency after reconnection.

  • The EOD process was not performing a garbage collection after a batch ingest.

  • Data was unqueryable after a batch ingest if no additional streaming data was loaded after the initial batch ingest.

Reliable Transport

  • In deployments with a one-node RT configuration, RT has been updated to roll its log file after a restart. This prevents data flow from being paused.The RT log file rolls from log.x.y to log.x.y+1.

Stream Processor

  • Previously, when using Python, 'sp.decode.csv' did not support reading columns as strings. This has been updated to enhance flexibility when reading table columns. It now accepts an empty string "" in lieu of a schema to read in all table columns as strings.

  • Sidecar containers no longer log at a TRACE level by default and now adhere to the log level that is set by the user.

  • The SP controller now adheres to the log level set by the user.

  • The CSV decoder did not support decoding csvs which had line-returns (i.e. \n’s) embedded in strings. This functionality can now be turned on/off with a new optional parameter called newlines. See .qsp.decode.csv for details.

  • Previously, when running a large import from a static source (file, Azure, AWS, GCP), there was no indication of progress in the logs. Now, periodic info is included in the SP logs while processing, indicating on-going progress of the import.

  • The SP’s status APIs now return TEARING_DOWN rather than NOTEXIST when a pipeline has PVCs that are still in a terminating state.

Security

  • Various CVEs have been remediated as part of this release.

Upgrade notes

  • The default image repository used for new 1.9 installations will be the KX Downloads Portal. The credentials to access this are stored in a docker config json secret named kxi-image-pull-secret. If you are upgrading from an earlier version that used the KX Nexus image repository with an exiting secret kxi-nexus-pull-secret, this existing repository and secret will be used.

  • After executing an upgrade, if an RT pod does not appear as ‘Ready’, the pod should be deleted/killed so that it is restarted.

  • During an upgrade, the RT pods attempt to delete a Kubernetes configmap. Since there were 3 RT pods, only one of the pods will be successful in deleting the config map. This means that the other pods will receive ERRORs back indicating that their delete attempt failed. These Failed to delete snapshot: tmp errors are benign and will be set to WARN messages in future.

  • Query Labels Style

    KXI_ALLOW_OLD_LABEL_STYLE now defaults to "false". It must be explicitly set to "true" in the RC to allow the "old label style". For example, in a system with labels region, and class:

    • Labels in queries (such as .kxi.getData) must be defined in the labels parameter.

      • old style: --data '{"table": "trade", "region": "usa", "class": "equities"}'
      • new style (new default): --data {"table": "trade", "labels": {"region": "usa", "class": "equities"} }'

    • Labels in SQL queries must be prefixed with label_.

      • old style: SELECT * FROM trade WHERE region='usa', class='equities'
      • new style (new default): SELECT * FROM trade WHERE label_region='usa', label_class='equities'

Known Issues

Expand to see the list of known issues here

Pipelines

  • When deploying a pipeline that writes to a table that does not exist, no error will be thrown.

Deprecation

  • prtnEndCB and reloadCB callback functions are being deprecated in 1.9.0 and may be removed from future releases.

  • RT maxLogSize is no longer used in configuration and will be ignored from older configurations. RT will now utilize all disk available to it.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

Stream Processor

microservice Nexus Docker Downloads Portal Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.9.0 portal.dl.kx.com/kxi-sp-controller:1.9.0
Worker registry.dl.kx.com/kxi-sp-worker:1.9.0 portal.dl.kx.com/kxi-sp-worker:1.9.0
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.9.0 portal.dl.kx.com/kxi-sp-coordinator:1.9.0
Worker-Python registry.dl.kx.com/kxi-sp-python:1.9.0 portal.dl.kx.com/kxi-sp-python:1.9.0
Worker-ML registry.dl.kx.com/kxi-ml:1.9.0 portal.dl.kx.com/kxi-ml:1.9.0

Database

microservice Nexus Docker Downloads Portal Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.9.0 portal.dl.kx.com/kxi-da:1.9.0
Single container registry.dl.kx.com/kxi-da-single:1.9.0 portal.dl.kx.com/kxi-da-single:1.9.0
Storage Manager registry.dl.kx.com/kxi-sm-single:1.9.0 portal.dl.kx.com/kxi-sm-single:1.9.0
Gateway registry.dl.kx.com/kxi-sg-gw:1.9.0 portal.dl.kx.com/kxi-sg-gw:1.9.0
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.9.0 portal.dl.kx.com/kxi-sg-rc:1.9.0
Aggregator registry.dl.kx.com/kxi-sg-agg:1.9.0 portal.dl.kx.com/kxi-sg-agg:1.9.0

Reliable Transport

microservice Nexus Docker Downloads Portal Docker Helm Chart
Reliable Transport registry.dl.kx.com/kxi-rt:1.9.0 portal.dl.kx.com/kxi-rt:1.9.0 kxi-rt-1.9.0.tgz

Miscellaneous

microservice Nexus Docker package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.9.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar