kdb Insights Microservices - Release Notes

This page details the release notes for the latest series of kdb Insights Microservices releases. To find the release notes for an older release, refer to the previous releases page.

1.7.1

Release Date

2023-09-19

Fixes

Expand to see the full detail of resolved issues here

Database

• Resolved an issue with the Storage Manager not rolling back aborted transactions if an EOD failed.

• Deleting data from the HDB using a filter option now correctly performs the delete. Previously, this returned an error claiming that the column(s) being filtered didn’t exist.

• EOIs are no longer delayed after an EOI recovery event. Previously, it was possible for EOIs to become held up if the process was terminated then subsequently recovered. The EOIs would still complete successfully, but with a delay.

Security

• Resolved CVE-2023-37920 Certifi's removal of e-Tugra root certificate, by upgrading the spaCy dependency.

Stream Processor

• Attempts to horizontally scale pipelines using the direct write feature will now explicitly error since this is not currently supported.

Artifacts

Service Discovery

microservice	Docker	Kubernetes
Discovery Registry	registry.dl.kx.com/kxi-eureka-discovery:1.7.0	https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.7.0.tgz
Discovery Proxy	registry.dl.kx.com/kxi-discovery-proxy:1.7.0

Stream Processor

microservice	Docker
Controller	registry.dl.kx.com/kxi-sp-controller:1.7.1
Worker	registry.dl.kx.com/kxi-sp-worker:1.7.1
Coordinator	registry.dl.kx.com/kxi-sp-coordinator:1.7.1
Worker-Python	registry.dl.kx.com/kxi-sp-python:1.7.1
Worker-ML	registry.dl.kx.com/kxi-ml:1.7.1

Database

microservice	Docker
Data Access Process (DA)	registry.dl.kx.com/kxi-da:1.7.1
Single container	registry.dl.kx.com/kxi-da-single:1.7.1
Storage Manager	registry.dl.kx.com/kxi-sm-single:1.7.1
Gateway	registry.dl.kx.com/kxi-sg-gw:1.7.1
Resource Coordinator	registry.dl.kx.com/kxi-sg-rc:1.7.1
Aggregator	registry.dl.kx.com/kxi-sg-agg:1.7.1

Machine Learning

microservice	package
ML Registry	https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-registry/0.8.0/kxi-ml-registry.0.8.0.qpk
ML Analytics	https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-analytics/0.8.0/kxi-ml-analytics.0.8.0.qpk

Reliable Transport

microservice	Docker	Kubernetes
Reliable Transport	registry.dl.kx.com/kxi-rt:1.7.0	https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-rt-1.7.0.tgz

Miscellaneous

microservice	Docker	package
Metrics Sidecar	registry.dl.kx.com/kxi-sidecar:1.7.0
OpenAPI SDK Generator	registry.dl.kx.com/kxi-openapi-codegen:0.8.0	https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

1.7.0

Release Date

2023-08-25

New Features

kdb Insights Database

• End-of-day writedown memory controls

  The Storage Manager (SM) process now supports restricting memory consumption for data writedown, giving the opportunity for cost savings. Reducing the RAM allocated to SM may cause EOD to run longer than if RAM was unrestricted.

  For guidance on tuning your RAM setup, see Performance best practices and the recommended system sizes for Azure here.

• Deleting data from the HDB

  The ability to perform targeted deletion of data from the HDB has been added. This lets you clean up unncessary or incorrect data without having to remove an entire partition where there may be data you need to maintain. There are a variety of ways to specify the data you wish to delete. See Database delete for further details.

• New Authorization Options

  Authorization capabilities have been extended to enable linking out to other authorization providers or to write your own. You can now set up a sidecar that offers either an HTTP or IPC interface for authorization to the Service Gateway of the database. See Custom HTTP Authorization for more details.

• Backfill a new column with derived data

  When adding a new column to your schema, you can now define a custom function that will backfill the new column with derived data.

• Publishing derived data from custom APIs

  Custom APIs that generate derived data can now stream and/or persist their output.

• Appending data on batch ingest

  When ingesting data using the direct write feature, you now have the option to append data rather than replace it. See overwrite option for more details.

Reliable Transport

• Flexible subscriptions to data streams

  RT supports the option to control at what point in a data stream you wish to subscribe. This new feature adds the ability for a subscriber(s) to subscribe to new data only. This is useful for subscriber(s) who do not need to receive historical data and just need to begin receiving data from the latest point onward. The API exposes information about what data is available to subscribers. This is available for use when subscribing using the C and q interface.

• C Subscriber Support in Reliable Transport

  The C language interface for RT is now able to subscribe to the data on an RT stream if the subscriber is inside the cluster. See here for more details. A new subscriber is now able to subscribe from the last message added to the stream as well as from the earliest message the stream holds.

Improvements

kdb Insights Database

• End-of-day writedown performance improvements

  The Storage Manager (SM) process has been optimized to significantly improve its performance - up to 50-70% reduction in EOD duration. This improved EOD performance can reduce the memory allocated or the time taken for writedown, depending on your workloads and configuration.

  For guidance and best practices on how to configure your schema and system to take advantage of these optimizations, review Performance best practices and the recommended system sizes for Azure here.

Stream Processor

• Readers

  • When using cloud and file readers, you can now use filename pattern matching to ingest multiple files using a single reader. This expands the ability for you to build up ingest workloads that pull in multiple files. For example, using an S3 reader, you could read all CSV files in all folders within a bucket using the pattern:

    s3://path/to/bucket/**/*.csv
    

  • When using cloud and file readers you can now use a file watcher which is usually paired with filename pattern matching described above. The reader will continually watch for new files matching the file path(s) provided. See here for details on:

    • q APIs: .qsp.read.fromGoogleStorage, .qsp.read.fromAzureStorage and .qsp.read.fromAmazonS3.
    • Python APIs: .qsp.read.fromGoogleStorage, from_azure_storage and from_amazon_s3.
    • UI: Google Cloud Storage, Microsoft Azure Storage and Amazon S3.

    Not to be used with the Write Direct to HDB option

    The Watcher options should not be used when also choosing the Write Direct to HDB option in the Database Writer in the UI, or the q or Python APIs.

Fixes

Expand to see the full detail of resolved issues here

kdb Insights Database

• Fixed an issue where schema type conversions from strings to GUIDs would fail and cause a type error.

• Fixed an issue on certain NFS-based system (namely Lustre) where the ‘current’ database pointer could get converted to a directory resulting in an unexpected error.

EISDIR: /data/idb/current.new -> /data/idb/current

• An error has been fixed that would occur when updating an object storage tier that used an inventory file after a data migration.

• Fixed an issue when querying with .kxi.getData using a groupBy operation on a foreign key column where the name of the foreign key column would be thrown and the query would fail to complete.

• A new warning has been added to the Resource Coordinator if no Aggregators connect to it. After a wait period, if an Aggregator continues not to connect, this will be escalated to an error to indicate that the system is not ready for queries.

• When the kdb Insights Database was configured with an object storage tier but an inventory file was not provided, the Storage Manager would error on startup with invalid object storage URI error. This has been changed to not attempt to create an inventory file if no configuration is explicitly provided.

• Resolved an issue when attempting to migrate HDB data to Azure Blob Storage. Previously, this would fail with a signature error.

• Fixed a potential race condition between NFS replication and DAP reload signals. It was previously possible that a DAP may reload before NFS replicated data which could result in data gaps. A new sequencing mechanism has been added to ensure DAPs do not reload until replication is complete.

• Values in a symbol column will now be properly available immediately after a batch ingest. Previously, these would be missing until the next EOI.

• Renaming a sort column no longer errors.

Artifacts

Service Discovery

microservice	Docker	Kubernetes
Discovery Registry	registry.dl.kx.com/kxi-eureka-discovery:1.7.0	https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.7.0.tgz
Discovery Proxy	registry.dl.kx.com/kxi-discovery-proxy:1.7.0

Stream Processor

microservice	Docker
Controller	registry.dl.kx.com/kxi-sp-controller:1.7.0
Worker	registry.dl.kx.com/kxi-sp-worker:1.7.0
Coordinator	registry.dl.kx.com/kxi-sp-coordinator:1.7.0
Worker-Python	registry.dl.kx.com/kxi-sp-python:1.7.0
Worker-ML	registry.dl.kx.com/kxi-ml:1.7.0

Database

microservice	Docker
Data Access Process (DA)	registry.dl.kx.com/kxi-da:1.7.0
Single container	registry.dl.kx.com/kxi-da-single:1.7.0
Storage Manager	registry.dl.kx.com/kxi-sm-single:1.7.0
Gateway	registry.dl.kx.com/kxi-sg-gw:1.7.0
Resource Coordinator	registry.dl.kx.com/kxi-sg-rc:1.7.0
Aggregator	registry.dl.kx.com/kxi-sg-agg:1.7.0

Machine Learning

microservice	package
ML Registry	https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-registry/0.8.0/kxi-ml-registry.0.8.0.qpk
ML Analytics	https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-analytics/0.8.0/kxi-ml-analytics.0.8.0.qpk

Reliable Transport

microservice	Docker	Kubernetes
Reliable Transport	registry.dl.kx.com/kxi-rt:1.7.0	https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-rt-1.7.0.tgz

Miscellaneous

microservice	Docker	package
Metrics Sidecar	registry.dl.kx.com/kxi-sidecar:1.7.1
OpenAPI SDK Generator	registry.dl.kx.com/kxi-openapi-codegen:0.8.0	https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar