Skip to content

kdb Insights Microservices - Release Notes

This page details the release notes for the latest series of kdb Insights Microservices releases. To find the release notes for an older release, refer to the previous releases page.

1.8.3

Release Date

2024-02-06

Improvements

  • Python Cloud Readers

    Python cloud readers now support polling for new matches to a glob pattern with the watch parameter. For more information on the watch parameter, refer to the Readers documentation.

Artifacts

Stream Processor

Microservice Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.8.3
Worker registry.dl.kx.com/kxi-sp-worker:1.8.3
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.8.3
Worker-Python registry.dl.kx.com/kxi-sp-python:1.8.3
Worker-ML registry.dl.kx.com/kxi-ml:1.8.3

Database

Microservice Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.8.2
Single container registry.dl.kx.com/kxi-da-single:1.8.2
Storage Manager registry.dl.kx.com/kxi-sm-single:1.8.2
Gateway registry.dl.kx.com/kxi-sg-gw:1.8.2
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.8.2
Aggregator registry.dl.kx.com/kxi-sg-agg:1.8.2

Reliable Transport

Microservice Docker Kubernetes
Reliable Transport registry.dl.kx.com/kxi-rt:1.8.0 https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-rt-1.8.0.tgz

Miscellaneous

Microservice Docker Package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.8.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

1.8.2

Release Date

2024-01-23

Fixes

Expand to see the full detail of resolved issues here

Database

  • An issue has been resolved in relation to Batch Ingest. Previously, some data in the Historical Database (HDB) was persisted to disk but it was unqueryable. This has been fixed, and all data ingested into the HDB is now queryable.

Artifacts

Stream Processor

Microservice Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.8.1
Worker registry.dl.kx.com/kxi-sp-worker:1.8.1
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.8.1
Worker-Python registry.dl.kx.com/kxi-sp-python:1.8.1
Worker-ML registry.dl.kx.com/kxi-ml:1.8.1

Database

Microservice Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.8.2
Single container registry.dl.kx.com/kxi-da-single:1.8.2
Storage Manager registry.dl.kx.com/kxi-sm-single:1.8.2
Gateway registry.dl.kx.com/kxi-sg-gw:1.8.2
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.8.2
Aggregator registry.dl.kx.com/kxi-sg-agg:1.8.2

Reliable Transport

Microservice Docker Kubernetes
Reliable Transport registry.dl.kx.com/kxi-rt:1.8.0 https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-rt-1.8.0.tgz

Miscellaneous

Microservice Docker Package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.8.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

1.8.1

Release Date

2024-01-16

Improvements

  • Stream Processer CSV Decoder Improvements

    The Python CSV decoder now supports the first header option. This should be used when only the first batch of your dataset contains a header in the CSV file. See Decoders for information on the CSV decoder configuration.

Fixes

Expand to see the full detail of resolved issues here

Stream Processor

  • Stream Processor Beta features configuration has been fixed. When deploying the SP Helm chart, setting betaFeatures: true wasn't correctly enabling Beta features.

  • Previously, the CSV decoder could drop messages when paired with a file reader (Amazon S3, Google Storage, Azure Storage) where chunking was enabled and the CSV decoder header was set to always. This issue has been resolved. See Decoders for information on the CSV decoder configuration.

  • SP Cloud Readers no longer error when there are no files matching the expected pattern. Previously, a cloud reader using glob pattern matching would prevent the pipeline from starting if no files matched. In particular, this would occur when trying to use the file watcher feature and no files matched as of the pipeline starting.

  • The delimiter select field in the Stream Processor CSV node has been updated to allow any delimiter to be used. Previously, there was a restriction on the number of characters that could be submitted, preventing entry of \t or any other delimiter.

  • Resolved the problem of all window nodes triggering their jobs at the same time, despite their configuration settings. This would occur in setups with pipelines with multiple window nodes, or separate pipelines that read and write to same RT stream. See Windows for more details.

Security

  • Various CVEs were remediated as part of this release.

Artifacts

Stream Processor

microservice Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.8.1
Worker registry.dl.kx.com/kxi-sp-worker:1.8.1
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.8.1
Worker-Python registry.dl.kx.com/kxi-sp-python:1.8.1
Worker-ML registry.dl.kx.com/kxi-ml:1.8.1

Database

microservice Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.8.0
Single container registry.dl.kx.com/kxi-da-single:1.8.0
Storage Manager registry.dl.kx.com/kxi-sm-single:1.8.0
Gateway registry.dl.kx.com/kxi-sg-gw:1.8.0
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.8.0
Aggregator registry.dl.kx.com/kxi-sg-agg:1.8.0

Reliable Transport

microservice Docker Kubernetes
Reliable Transport registry.dl.kx.com/kxi-rt:1.8.0 https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-rt-1.8.0.tgz

Miscellaneous

microservice Docker package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.8.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

1.8.0

Release Date

2023-12-11

New Features

  • Streaming via WebSocket

    You can now stream from a pipeline to a client application via WebSocket. This supports the ability to stream both snapshots as well as real-time data. See here for a tutorial on using this feature.

  • Azure and GCP Parquet Reader Support

    The kdb Insights Parquet Reader functionality has been extended to support Azure and GCP object storage. Users can now read, decode, and watch Parquet files stored on Azure and GCP, as well as AWS. These enhancements are available in the q API and the Python API.

  • HTTP Publishing

    kdb Insights has introduced a REST interface to publish data over HTTP. This expands the interoperability of kdb Insights for ingesting data. Note that file size is limited to 10MB per publish. See Upload via HTTP for more details.

  • Python Interface to Reliable Transport

    A new Python interface allows you to publish data to Reliable Transport from your own Python application in Docker, or on Linux or Windows. See here for details.

  • New Database Preview API

    You can now query a sample of your data from the database to understand your data/schema, construct more complex queries, or build visuals on the full data set. See here for details.

  • Nested Filters in Queries

    You can now express a complex filter in a getData query using nested filters. See here for details.

Improvements

  • Autoscaling Service Gateways

    The Service Gateway and Aggregator can now autoscale if CPU exceeds your configured threshold. The default is to trigger scaling when CPU exceeds 80% for 5 seconds, with a default stabilization period of 300 seconds. All of this is configurable in your values file. See here for details.

  • Stream Processor File Watcher

    Cloud file readers in the Stream Processor now support watching for new files. You can now provide a pattern expression to search for file names. When watching is enabled, the file reader will poll for new files that match the expression. For full details on file watching, see the Amazon S3 reader, the Azure Blob storage reader and the Google Cloud storage reader.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Improved startup performance of HDB DAPs when loading late data from IDB tier mount.

  • Storage Manager resiliency improvements have been made to improve how SM cleans up aborted transactions after a failure event.

  • Improved failure recovery for object storage in Storage Manager. This resolves a case where EOI would not complete and would report SM Flushing filesystem with [directory] errors.

Reliable Transport

  • Logging for LEADER/FOLLOWER updates have been changed to DEBUG level to reduce log noise.

Stream Processor

  • Previously, it was possible that FINISHED SP pipelines using multiple partitioned workers could spawn extra zombie workers. This has been resolved.

  • Pipeline quick tests no longer fail if they contained a kdb Insights Stream or kdb Insights Database reader or writer.

  • Previously it was not possible to maintain state of any object that contained a kdb guid type. This is now fixed. However, if a rollback to an earlier version is required after an upgrade, a rollback step should be run so that on-disk checkpoints can be converted to a format compatible with earlier versions.

  • Fixed a bug where the csv decoder, with option header=first, and paired with an unbounded streaming reader, was incorrectly dropping the first record in every stream batch.

  • Fixed a determinism gap when reading objects from cloud storage using a glob pattern. In a recovery scenario, it was possible to receive duplicate data.

  • Fixed a bug where unpopulated column of type 0h (i.e. list) was not getting filled in with blank entries correctly.

  • If a pipeline is deployed as part of an assembly, .qsp.write.toDatabase now defaults to writing to that assembly. This means the assembly name can be omitted when calling .qsp.write.toDatabase, for example, .qsp.write.toDatabase[trade; .qsp.use`directWrite!(::;1b)].

  • Fixed a bug where in a single path pipeline, the existence of more than one emitter resulted in that same number of combine nodes being added.

File Watchers

  • When using the file/cloud readers in a non-file watching scenario, files that were not getting chunked did not have their state checkpointed. This meant that during a recovery scenario, duplicate data would be generated for those files.

  • An issue has been resolved that could cause the file watcher to lose data at the boundaries when chunking was set to auto.

Security

  • Various CVEs were remediated as part of this release.

Artifacts

Stream Processor

microservice Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.8.0
Worker registry.dl.kx.com/kxi-sp-worker:1.8.0
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.8.0
Worker-Python registry.dl.kx.com/kxi-sp-python:1.8.0
Worker-ML registry.dl.kx.com/kxi-ml:1.8.0

Database

microservice Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.8.0
Single container registry.dl.kx.com/kxi-da-single:1.8.0
Storage Manager registry.dl.kx.com/kxi-sm-single:1.8.0
Gateway registry.dl.kx.com/kxi-sg-gw:1.8.0
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.8.0
Aggregator registry.dl.kx.com/kxi-sg-agg:1.8.0

Reliable Transport

microservice Docker Kubernetes
Reliable Transport registry.dl.kx.com/kxi-rt:1.8.0 https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-rt-1.8.0.tgz

Miscellaneous

microservice Docker package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.8.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar