kdb Insights Microservices - Release Notes
This page details the release notes for the latest series of kdb Insights Microservices releases. To find the release notes for an older release, refer to the previous releases page.
1.9.4
Release Date
2024-06-11
Fixes
Expand to see the full detail of resolved issues here
Security
- Multiple CVEs have been resolved as part of this patch.
Cloud Readers
-
An issue has been resolved where, when administering a pipeline, a cloud reader node (s3, ms, and gs) could prompt you that there were unsaved changes even when no changes have been made.
-
The Azure reader file watcher was returning an error that a file(s) did not exist despite the file(s) being present. This has now been fixed.
Reliable Transport
- RT's replicator was emitting an
Invalid publishing directory, was it deleted by mistake? Please enrol a different client and restart the SDK with the new clienterror after recovering from a power outage of a q publisher server. This has been resolved to handle the recovery cleanly after such an outage.
Database
- For a new system with an empty database, queries would not work after the first end-of-day data writedown if no data had been ingested yet. This issue has been resolved.
Artifacts
KX Nexus
The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.
Stream Processor
| microservice | Nexus Docker | Downloads Portal Docker |
|---|---|---|
| Controller | registry.dl.kx.com/kxi-sp-controller:1.9.4 | portal.dl.kx.com/kxi-sp-controller:1.9.4 |
| Worker | registry.dl.kx.com/kxi-sp-worker:1.9.4 | portal.dl.kx.com/kxi-sp-worker:1.9.4 |
| Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.9.4 | portal.dl.kx.com/kxi-sp-coordinator:1.9.4 |
| Worker-Python | registry.dl.kx.com/kxi-sp-python:1.9.4 | portal.dl.kx.com/kxi-sp-python:1.9.4 |
| Worker-ML | registry.dl.kx.com/kxi-ml:1.9.4 | portal.dl.kx.com/kxi-ml:1.9.4 |
Database
| microservice | Nexus Docker | Downloads Portal Docker |
|---|---|---|
| Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.9.4 | portal.dl.kx.com/kxi-da:1.9.4 |
| Single container | registry.dl.kx.com/kxi-da-single:1.9.4 | portal.dl.kx.com/kxi-da-single:1.9.4 |
| Storage Manager | registry.dl.kx.com/kxi-sm-single:1.9.4 | portal.dl.kx.com/kxi-sm-single:1.9.4 |
| Gateway | registry.dl.kx.com/kxi-sg-gw:1.9.4 | portal.dl.kx.com/kxi-sg-gw:1.9.4 |
| Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.9.4 | portal.dl.kx.com/kxi-sg-rc:1.9.4 |
| Aggregator | registry.dl.kx.com/kxi-sg-agg:1.9.4 | portal.dl.kx.com/kxi-sg-agg:1.9.4 |
| Rollback | registry.dl.kx.com/kxi-db-rollback:1.9.4 | portal.dl.kx.com/kxi-db-rollback:1.9.4 |
Reliable Transport
| microservice | Nexus Docker | Downloads Portal Docker | Helm Chart |
|---|---|---|---|
| Reliable Transport | registry.dl.kx.com/kxi-rt:1.9.1 | portal.dl.kx.com/kxi-rt:1.9.1 | kxi-rt-1.9.1.tgz |
Miscellaneous
| microservice | Nexus Docker | package |
|---|---|---|
| Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.9.0 | |
| OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.9.3
Release Date
2024-05-21
Fixes
Expand to see the full detail of resolved issues here
Service Gateway
- The
KXI_RC_LABEL_SELECTORfield was not being correctly honored, causing Kubernetes discovery to become non-functional. This has been resolved. Refer to Gateway for details on this field.
Stream Processor
- The SP Kafka writer did not support configuration options when connecting to a Kafka broker. An
optionsparameter has been added to the q & Python APIs to allow the client to specify advanced configuration, for example, authentication or TLS. See the Writers documentation for q and Python for details.
Artifacts
KX Nexus
The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.
Stream Processor
| microservice | Nexus Docker | Downloads Portal Docker |
|---|---|---|
| Controller | registry.dl.kx.com/kxi-sp-controller:1.9.3 | portal.dl.kx.com/kxi-sp-controller:1.9.3 |
| Worker | registry.dl.kx.com/kxi-sp-worker:1.9.3 | portal.dl.kx.com/kxi-sp-worker:1.9.3 |
| Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.9.3 | portal.dl.kx.com/kxi-sp-coordinator:1.9.3 |
| Worker-Python | registry.dl.kx.com/kxi-sp-python:1.9.3 | portal.dl.kx.com/kxi-sp-python:1.9.3 |
| Worker-ML | registry.dl.kx.com/kxi-ml:1.9.3 | portal.dl.kx.com/kxi-ml:1.9.3 |
Database
| microservice | Nexus Docker | Downloads Portal Docker |
|---|---|---|
| Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.9.3 | portal.dl.kx.com/kxi-da:1.9.3 |
| Single container | registry.dl.kx.com/kxi-da-single:1.9.3 | portal.dl.kx.com/kxi-da-single:1.9.3 |
| Storage Manager | registry.dl.kx.com/kxi-sm-single:1.9.3 | portal.dl.kx.com/kxi-sm-single:1.9.3 |
| Gateway | registry.dl.kx.com/kxi-sg-gw:1.9.3 | portal.dl.kx.com/kxi-sg-gw:1.9.3 |
| Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.9.3 | portal.dl.kx.com/kxi-sg-rc:1.9.3 |
| Aggregator | registry.dl.kx.com/kxi-sg-agg:1.9.3 | portal.dl.kx.com/kxi-sg-agg:1.9.3 |
| Rollback | registry.dl.kx.com/kxi-db-rollback:1.9.3 | portal.dl.kx.com/kxi-db-rollback:1.9.3 |
Reliable Transport
| microservice | Nexus Docker | Downloads Portal Docker | Helm Chart |
|---|---|---|---|
| Reliable Transport | registry.dl.kx.com/kxi-rt:1.9.1 | portal.dl.kx.com/kxi-rt:1.9.1 | kxi-rt-1.9.1.tgz |
Miscellaneous
| microservice | Nexus Docker | package |
|---|---|---|
| Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.9.0 | |
| OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.9.2
Release Date
2024-05-07
Fixes
Expand to see the full detail of resolved issues here
Reliable Transport
- The RT client was failing to recognize a new log file, putting downstream subscribers into a hung state. This has been resolved.
Artifacts
KX Nexus
The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.
Stream Processor
| microservice | Nexus Docker | Downloads Portal Docker |
|---|---|---|
| Controller | registry.dl.kx.com/kxi-sp-controller:1.9.2 | portal.dl.kx.com/kxi-sp-controller:1.9.2 |
| Worker | registry.dl.kx.com/kxi-sp-worker:1.9.2 | portal.dl.kx.com/kxi-sp-worker:1.9.2 |
| Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.9.2 | portal.dl.kx.com/kxi-sp-coordinator:1.9.2 |
| Worker-Python | registry.dl.kx.com/kxi-sp-python:1.9.2 | portal.dl.kx.com/kxi-sp-python:1.9.2 |
| Worker-ML | registry.dl.kx.com/kxi-ml:1.9.2 | portal.dl.kx.com/kxi-ml:1.9.2 |
Database
| microservice | Nexus Docker | Downloads Portal Docker |
|---|---|---|
| Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.9.2 | portal.dl.kx.com/kxi-da:1.9.2 |
| Single container | registry.dl.kx.com/kxi-da-single:1.9.2 | portal.dl.kx.com/kxi-da-single:1.9.2 |
| Storage Manager | registry.dl.kx.com/kxi-sm-single:1.9.2 | portal.dl.kx.com/kxi-sm-single:1.9.2 |
| Gateway | registry.dl.kx.com/kxi-sg-gw:1.9.2 | portal.dl.kx.com/kxi-sg-gw:1.9.2 |
| Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.9.2 | portal.dl.kx.com/kxi-sg-rc:1.9.2 |
| Aggregator | registry.dl.kx.com/kxi-sg-agg:1.9.2 | portal.dl.kx.com/kxi-sg-agg:1.9.2 |
| Rollback | registry.dl.kx.com/kxi-db-rollback:1.9.2 | portal.dl.kx.com/kxi-db-rollback:1.9.2 |
Reliable Transport
| microservice | Nexus Docker | Downloads Portal Docker | Helm Chart |
|---|---|---|---|
| Reliable Transport | registry.dl.kx.com/kxi-rt:1.9.1 | portal.dl.kx.com/kxi-rt:1.9.1 | kxi-rt-1.9.1.tgz |
Miscellaneous
| microservice | Nexus Docker | package |
|---|---|---|
| Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.9.0 | |
| OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.9.1
Release Date
2024-04-30
Improvements
- The
fromParquetReader is now supported for AWS-based deployments. For more information, refer to Readers.
Fixes
Expand to see the full detail of resolved issues here
Database
- Client requests using compressed IPC messages are now properly serviced. Previously, if the first request from a client was using compressed IPC, that and any future requests would fail.
Reliable Transport
- If using the RT Helm chart, RT pods now use Kubernetes liveness probes to automatically bounce pods if unresponsive. This resolves a known issue in 1.9.0 in which RT pods were required to be manually restarted if an upgrade erroneously left them in a non-ready state.
Artifacts
KX Nexus
The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.
Stream Processor
| microservice | Nexus Docker | Downloads Portal Docker |
|---|---|---|
| Controller | registry.dl.kx.com/kxi-sp-controller:1.9.1 | portal.dl.kx.com/kxi-sp-controller:1.9.1 |
| Worker | registry.dl.kx.com/kxi-sp-worker:1.9.1 | portal.dl.kx.com/kxi-sp-worker:1.9.1 |
| Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.9.1 | portal.dl.kx.com/kxi-sp-coordinator:1.9.1 |
| Worker-Python | registry.dl.kx.com/kxi-sp-python:1.9.1 | portal.dl.kx.com/kxi-sp-python:1.9.1 |
| Worker-ML | registry.dl.kx.com/kxi-ml:1.9.1 | portal.dl.kx.com/kxi-ml:1.9.1 |
Database
| microservice | Nexus Docker | Downloads Portal Docker |
|---|---|---|
| Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.9.1 | portal.dl.kx.com/kxi-da:1.9.1 |
| Single container | registry.dl.kx.com/kxi-da-single:1.9.1 | portal.dl.kx.com/kxi-da-single:1.9.1 |
| Storage Manager | registry.dl.kx.com/kxi-sm-single:1.9.1 | portal.dl.kx.com/kxi-sm-single:1.9.1 |
| Gateway | registry.dl.kx.com/kxi-sg-gw:1.9.1 | portal.dl.kx.com/kxi-sg-gw:1.9.1 |
| Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.9.1 | portal.dl.kx.com/kxi-sg-rc:1.9.1 |
| Aggregator | registry.dl.kx.com/kxi-sg-agg:1.9.1 | portal.dl.kx.com/kxi-sg-agg:1.9.1 |
| Rollback | registry.dl.kx.com/kxi-db-rollback:1.9.1 | portal.dl.kx.com/kxi-db-rollback:1.9.1 |
Reliable Transport
| microservice | Nexus Docker | Downloads Portal Docker | Helm Chart |
|---|---|---|---|
| Reliable Transport | registry.dl.kx.com/kxi-rt:1.9.1 | portal.dl.kx.com/kxi-rt:1.9.1 | kxi-rt-1.9.1.tgz |
Miscellaneous
| microservice | Nexus Docker | package |
|---|---|---|
| Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.9.0 | |
| OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.9.0
Release Date
2024-03-26
New Features
-
SQL Enhancements
SQL in kdb Insights now includes support for ORDER BY and LIMIT clauses, in addition to other functions. Changes to how SQL statements are parsed also significantly improves SQL query performance in this release. Query performance, on average, is up to 3x faster than before. Refer to SQL 2 for more information on the new SQL implementation and its current Known Limitations.
-
Snapshots & Majority Archival for Reliable Transport
Reliable Transport now supports taking snapshots allowing for majority archival of logs files. Previously, all nodes needed to be healthy before RT logs could be archived.
-
Configurable Connections for Reliable Transport
Reliable Transport now supports subscribers and publishers setting customized parameters when they register with their respective sequencer cluster. This gives you greater configurability in how you deploy and orchestrate components of RT. See rt.qpk for more details.
Improvements
-
Batch Delete
Batch delete now enables you to delete from all tiers of your HDB. Previously, this was only possible for the tier with the most recent HDB data. See Database Delete for details on how to perform a delete.
-
Python Cloud Readers
Python cloud readers now support polling for new matches to a glob pattern with the
watchparameter. See Readers for details. -
Upgrade to Insights Core 4.1.1
kdb Insights images have been upgraded to run Insights Core 4.1.1, which runs kdb+ 4.1 2024.03.12.
-
Upgrade to RockyLinux 9
kdb Insights images have been upgraded to run on Rocky Linux version 9 as their base operating system. This addresses security vulnerabilities present in version 8.
Fixes
Expand to see the full detail of resolved issues here
Database
-
If EOD failed while in the middle of an HDB transaction, the aborted transaction was not properly cleaned up after the failure and could result in data inconsistency after reconnection.
-
The EOD process was not performing a garbage collection after a batch ingest.
-
Data was unqueryable after a batch ingest if no additional streaming data was loaded after the initial batch ingest.
Reliable Transport
- In deployments with a one-node RT configuration, RT has been updated to roll its log file after a restart. This prevents data flow from being paused.The RT log file rolls from log.x.y to log.x.y+1.
Stream Processor
-
Previously, when using Python, 'sp.decode.csv' did not support reading columns as strings. This has been updated to enhance flexibility when reading table columns. It now accepts an empty string "" in lieu of a schema to read in all table columns as strings.
-
Sidecar containers no longer log at a
TRACElevel by default and now adhere to the log level that is set by the user. -
The SP controller now adheres to the log level set by the user.
-
The CSV decoder did not support decoding csvs which had line-returns (i.e. \n’s) embedded in strings. This functionality can now be turned on/off with a new optional parameter called
newlines. See .qsp.decode.csv for details. -
Previously, when running a large import from a static source (file, Azure, AWS, GCP), there was no indication of progress in the logs. Now, periodic info is included in the SP logs while processing, indicating on-going progress of the import.
-
The SP’s status APIs now return
TEARING_DOWNrather thanNOTEXISTwhen a pipeline has PVCs that are still in a terminating state.
Security
- Various CVEs have been remediated as part of this release.
Upgrade notes
-
The default image repository used for new 1.9 installations will be the KX Downloads Portal. The credentials to access this are stored in a docker config json secret named
kxi-image-pull-secret. If you are upgrading from an earlier version that used the KX Nexus image repository with an exiting secretkxi-nexus-pull-secret, this existing repository and secret will be used. -
After executing an upgrade, if an RT pod does not appear as ‘Ready’, the pod should be deleted/killed so that it is restarted.
-
During an upgrade, the RT pods attempt to delete a Kubernetes configmap. Since there were 3 RT pods, only one of the pods will be successful in deleting the config map. This means that the other pods will receive ERRORs back indicating that their delete attempt failed. These
Failed to delete snapshot: tmperrors are benign and will be set to WARN messages in future. -
Query Labels Style
KXI_ALLOW_OLD_LABEL_STYLEnow defaults to "false". It must be explicitly set to "true" in the RC to allow the "old label style". For example, in a system with labelsregion, andclass:-
Labels in queries (such as
.kxi.getData) must be defined in thelabelsparameter.- old style:
--data '{"table": "trade", "region": "usa", "class": "equities"}' - new style (new default):
--data {"table": "trade", "labels": {"region": "usa", "class": "equities"} }'
- old style:
-
Labels in SQL queries must be prefixed with
label_.- old style:
SELECT * FROM trade WHERE region='usa', class='equities' - new style (new default):
SELECT * FROM trade WHERE label_region='usa', label_class='equities'
- old style:
-
Known Issues
Expand to see the list of known issues here
Pipelines
- When deploying a pipeline that writes to a table that does not exist, no error will be thrown.
Deprecation
-
prtnEndCBandreloadCBcallback functions are being deprecated in 1.9.0 and may be removed from future releases. -
RT
maxLogSizeis no longer used in configuration and will be ignored from older configurations. RT will now utilize all disk available to it.
Artifacts
KX Nexus
The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.
Stream Processor
| microservice | Nexus Docker | Downloads Portal Docker |
|---|---|---|
| Controller | registry.dl.kx.com/kxi-sp-controller:1.9.0 | portal.dl.kx.com/kxi-sp-controller:1.9.0 |
| Worker | registry.dl.kx.com/kxi-sp-worker:1.9.0 | portal.dl.kx.com/kxi-sp-worker:1.9.0 |
| Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.9.0 | portal.dl.kx.com/kxi-sp-coordinator:1.9.0 |
| Worker-Python | registry.dl.kx.com/kxi-sp-python:1.9.0 | portal.dl.kx.com/kxi-sp-python:1.9.0 |
| Worker-ML | registry.dl.kx.com/kxi-ml:1.9.0 | portal.dl.kx.com/kxi-ml:1.9.0 |
Database
| microservice | Nexus Docker | Downloads Portal Docker |
|---|---|---|
| Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.9.0 | portal.dl.kx.com/kxi-da:1.9.0 |
| Single container | registry.dl.kx.com/kxi-da-single:1.9.0 | portal.dl.kx.com/kxi-da-single:1.9.0 |
| Storage Manager | registry.dl.kx.com/kxi-sm-single:1.9.0 | portal.dl.kx.com/kxi-sm-single:1.9.0 |
| Gateway | registry.dl.kx.com/kxi-sg-gw:1.9.0 | portal.dl.kx.com/kxi-sg-gw:1.9.0 |
| Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.9.0 | portal.dl.kx.com/kxi-sg-rc:1.9.0 |
| Aggregator | registry.dl.kx.com/kxi-sg-agg:1.9.0 | portal.dl.kx.com/kxi-sg-agg:1.9.0 |
| Rollback | registry.dl.kx.com/kxi-db-rollback:1.9.0 | portal.dl.kx.com/kxi-db-rollback:1.9.0 |
Reliable Transport
| microservice | Nexus Docker | Downloads Portal Docker | Helm Chart |
|---|---|---|---|
| Reliable Transport | registry.dl.kx.com/kxi-rt:1.9.0 | portal.dl.kx.com/kxi-rt:1.9.0 | kxi-rt-1.9.0.tgz |
Miscellaneous
| microservice | Nexus Docker | package |
|---|---|---|
| Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.9.0 | |
| OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |