Skip to content

kdb Insights Microservices - Release Notes

This page details the release notes for the latest series of kdb Insights Microservice releases. To find the release notes for an older release, refer to the previous releases page.

1.5.2

Release Date

2023-05-31

Improvements

  • The getData API can now perform aggregations on foreign key columns.

Fixes

Expand to see the full detail of resolved issues here

Stream Processor

  • Pipelines can be now be safely torn down without checkpoints having been cleared. Previously, if a teardown was followed by a restart, it is possible the pipeline would never run.

Artifacts

Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.

Service Discovery

microservice Docker Kubernetes
Discovery Registry registry.dl.kx.com/kxi-eureka-discovery:1.5.1 https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.5.1.tgz
Discovery Proxy registry.dl.kx.com/kxi-discovery-proxy:1.5.1

Stream Processor

microservice Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.5.2
Worker registry.dl.kx.com/kxi-sp-worker:1.5.2
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.5.2
Worker-Python registry.dl.kx.com/kxi-sp-python:1.5.2
Worker-ML registry.dl.kx.com/kxi-ml:1.5.2

Data Access

microservice Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.5.2
Single container registry.dl.kx.com/kxi-da-single:1.5.2

Storage Manager

microservice Docker
Storage Manager registry.dl.kx.com/kxi-sm:1.52
End of Interval (EOI) registry.dl.kx.com/kxi-sm-eoi:1.5.2
End of Day (EOD) registry.dl.kx.com/kxi-sm-eod:1.5.2
Database Migration (DBM) registry.dl.kx.com/kxi-sm-dbm:1.5.2
Single Container registry.dl.kx.com/kxi-sm-single:1.5.2

Service Gateway

microservice Docker
Gateway registry.dl.kx.com/kxi-sg-gw:1.5.2
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.5.2
Aggregator registry.dl.kx.com/kxi-sg-agg:1.5.2

Machine Learning

microservice package
ML Registry https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-registry/0.8.0/kxi-ml-registry.0.8.0.qpk
ML Analytics https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-analytics/0.8.0/kxi-ml-analytics.0.8.0.qpk

Reliable Transport

microservice Docker Kubernetes
Reliable Transport registry.dl.kx.com/kxi-rt:1.5.1 https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-rt-1.5.1.tgz

Miscellaneous

microservice Docker package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.5.1
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

1.5.1

Release Date

2023-05-19

Improvements

kdb Insights Database

  • SQL queries that filter by time but do not explicitly select the time column are now supported.

  • Query timeout messages now indicate a reason for the timeout. Previously, query timeouts would give no reason or say “timed out for unspecified reasons”. See queueing timeouts for more details.

Fixes

Expand to see the full detail of resolved issues here

kdb Insights Database

  • Databases without object storage tiers will no longer error and restart to resume normal operations after a schema conversion is performed.

  • New validation has been added to ensure that the snap value set in the EOD schedule of an assembly is always a time value. Previously, some examples showed usage of this field with a timestamp which was incorrectly ignored.

  • Sidecar containers are no longer automatically registering to object storage IAM service accounts where available. With large deployments, this could result in causing authentication backoffs for more mission critical processes.

  • The kxi_da_request_time metric has been changed to use a histogram type to reduce metric output. This was previously a summary type.

  • A new connection heartbeat has been added from DAPs to RC to improve connection stability when an RC restarts and is scheduled with a new IP.

Artifacts

Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.

Service Discovery

microservice Docker Kubernetes
Discovery Registry registry.dl.kx.com/kxi-eureka-discovery:1.5.1 https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.5.1.tgz
Discovery Proxy registry.dl.kx.com/kxi-discovery-proxy:1.5.1

Stream Processor

microservice Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.5.1
Worker registry.dl.kx.com/kxi-sp-worker:1.5.1
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.5.1
Worker-Python registry.dl.kx.com/kxi-sp-python:1.5.1
Worker-ML registry.dl.kx.com/kxi-ml:1.5.1

Data Access

microservice Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.5.1
Single container registry.dl.kx.com/kxi-da-single:1.5.1

Storage Manager

microservice Docker
Storage Manager registry.dl.kx.com/kxi-sm:1.5.1
End of Interval (EOI) registry.dl.kx.com/kxi-sm-eoi:1.5.1
End of Day (EOD) registry.dl.kx.com/kxi-sm-eod:1.5.1
Database Migration (DBM) registry.dl.kx.com/kxi-sm-dbm:1.5.1
Single Container registry.dl.kx.com/kxi-sm-single:1.5.1

Service Gateway

microservice Docker
Gateway registry.dl.kx.com/kxi-sg-gw:1.5.1
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.5.1
Aggregator registry.dl.kx.com/kxi-sg-agg:1.5.1

Machine Learning

microservice package
ML Registry https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-registry/0.8.0/kxi-ml-registry.0.8.0.qpk
ML Analytics https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-analytics/0.8.0/kxi-ml-analytics.0.8.0.qpk

Reliable Transport

microservice Docker Kubernetes
Reliable Transport registry.dl.kx.com/kxi-rt:1.5.1 https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-rt-1.5.1.tgz

Miscellaneous

microservice Docker package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.5.1
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

1.5.0

Release Date

2023-05-09

[NEW] New Features

kdb Insights Database

  • The kdb Insights Database now supports a REST interface for loading historical datasets into an existing database. You can now perform a historical batch ingest using a simple REST API. For more details, see batch ingestion.

  • To protect against reserved word and column name conflicts, the way labels are passed to a query has changed. Labels should now be specified either as a dictionary, or with the prefix label_. The old label style is still supported in 1.5, but is deprecated and will be disabled by default in 1.6. See upgrade considerations on labels for more details.

  • Distinct has been added to the available aggregations in getData for database queries.

Stream Processor

  • The Stream Processor Python API can now leverage automatic schema conversion using a new function kxi.sp.transform.schema.

  • Users can now use the Kafka Schema Registry in Python using kxi.sp.read.from_kafka.

  • String manipulation utilities have been added to the Stream Processor Python and q APIs, allowing users to selectively convert the string data to upper or lowercase representations for the purpose of data cleansing.

  • Pipeline deployments now support pipeline replicas, providing enhanced resilience to deployed pipelines. Users can now choose to run their pipeline with replicas for high availability in the Stream Processor. This replaces the previous use of pipeline groups.

Reliable Transport

  • Reliable Transport now offers subscribers the ability to use dynamic client-side filtering, enabling users to control what parts of the data feed their subscribers need to receive. This can greatly improve ingestion performance by eliminating the need for subscribers to handle discarding of unnecessary data. See Filtering for more information.

  • RT has introduced logHistory, a new API to assist in analysis and debugging. The API returns metadata about the RT log file history for logs that are no longer available. This includes:

    • The log file name.
    • The time of the first message added to that log file.
    • The position of the first message added to that log file. See Log File Meta Data for more information.

Improvements

kdb Insights Database

  • Batch ingest now supports ingesting tables that have no symbols. Additionally, extra validation has been added to verify symbol integrity.

  • Batch ingest now supports sorting of columns.

  • The kdb Insights Database gateway now supports returning responses that are greater than 2GB for IPC requests. See routing configuration for more details.

  • Database schemas in the UI can now be modified for assemblies that have been stopped. See schema modification.

  • Requests that supply a log correlator (logCorr) will now see that correlator in the Service Gateway logs. Previously a random correlator was used in the Service Gateway logs.

Fixes

Expand to see the full detail of resolved issues here

kdb Insights Database

  • Migrating tiers to object storage could fail with the eventual error SMDBM Migration discontinued due to failure if multiple threads were used.

  • New validation has been added to prevent accidentally adding a object storage tier to a database that does not already have one. This would previously error at EOD with a file not found error.

  • Schema conversions are now more robust on network based file systems (ex. Rook Ceph). Previously, this could result in a EXDEV. OS Reports: Invalid cross-device link error in the Storage Manager.

  • Changes to disk sort columns in a schema that would be incompatible with the current disk sort will now error with a clear message at deploy time. Previously, this would error with a u-fail error either at an EOI or EOD.

  • Batch ingest can now gracefully handle scenarios where there is no data provided in the session. Previously, this would result in the following error: Error reporting to licensing server: Value must be a long, leaving the ingest pipeline stuck in a RUNNING state.

  • When removing a symbol column from a table, the SM should be able to properly handle its removal as well as the sym file.

  • If an in-memory sort is applied to a column in a schema, it is now sorted on arrival in the RDB. Previously, if data arrived out of order with this set, an s-fail error would be thrown.

  • In the Storage Manager metrics, the kxi_sm_hdb_date_records metric no longer reports an empty value at startup, which resulted in false alerts about the HDB being offline.

  • The Storage Manager now double buffers its state table in order to protect against corruption.

  • The kxi_sg_pending metric would previously be decremented twice when a connection closed, resulting in negative pending values.

  • The Service Gateways were creating duplicate connections to Resource Coordinators while they were waiting to come online. This was resulting in a large number of redundant connections being opened between the services. In large deployments, this could result in conn errors in the Resource Coordinator.

  • RDBs and HDBs that require a dependency on the IDB will now error at startup if one is not specified as part of the mount configuration.

Stream Processor

  • The presence of a UTF-8 byte-order mark at the beginning of CSV files will now be automatically removed from data when acted on by the Stream Processor CSV decoder.

  • Window nodes in the Stream Processor can now accept dictionaries as input batches. Previously, incoming dictionaries would be merged with input batch in the buffer, instead of appending to a buffered list of dictionaries.

  • The Python write.to_database API previously converted assembly names to booleans. This is now resolved, and also fully supports the direct write functionality.

  • The default onFinish functions for the apply node have been updated to have the correct parameters and examples of flushing a buffer.

  • When using directWrite functionality in .qsp.write.toDatabase, table names as strings are now supported. Previously, this had to be a symbol.

  • Security vulnerabilities patched in the Stream Processor for the following CVEs: CVE-2022-41910, CVE-2022-41902, CVE-2022-41900, CVE-2022-41880.

Artifacts

Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.

Service Discovery

microservice Docker Kubernetes
Discovery Registry registry.dl.kx.com/kxi-eureka-discovery:1.5.0 https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.5.0.tgz
Discovery Proxy registry.dl.kx.com/kxi-discovery-proxy:1.5.0

Stream Processor

microservice Docker
Controller registry.dl.kx.com/kxi-sp-controller:1.5.0
Worker registry.dl.kx.com/kxi-sp-worker:1.5.0
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.5.0
Worker-Python registry.dl.kx.com/kxi-sp-python:1.5.0
Worker-ML registry.dl.kx.com/kxi-ml:1.5.0

Data Access

microservice Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.5.0
Single container registry.dl.kx.com/kxi-da-single:1.5.0

Storage Manager

microservice Docker
Storage Manager registry.dl.kx.com/kxi-sm:1.5.0
End of Interval (EOI) registry.dl.kx.com/kxi-sm-eoi:1.5.0
End of Day (EOD) registry.dl.kx.com/kxi-sm-eod:1.5.0
Database Migration (DBM) registry.dl.kx.com/kxi-sm-dbm:1.5.0
Single Container registry.dl.kx.com/kxi-sm-single:1.5.0

Service Gateway

microservice Docker
Gateway registry.dl.kx.com/kxi-sg-gw:1.5.0
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.5.0
Aggregator registry.dl.kx.com/kxi-sg-agg:1.5.0

Machine Learning

microservice package
ML Registry https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-registry/0.8.0/kxi-ml-registry.0.8.0.qpk
ML Analytics https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-analytics/0.8.0/kxi-ml-analytics.0.8.0.qpk

Reliable Transport

microservice Docker Kubernetes
Reliable Transport registry.dl.kx.com/kxi-rt:1.5.0 https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-rt-1.5.0.tgz

Miscellaneous

microservice Docker package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.5.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

    To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below

    klic license renew $LICENSE_UUID --version=latest
    
    env QLIC=$(pwd) klic license fetch $LICENSE_UUID
    

    Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.

    printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml
    
    kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml
    
  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.