Release Notes - kdb Insights Microservices 1.4.0

kdb Insights Microservices 1.4.0 release.

Release Date

2023-03-03

New Features

Reliable Transport

[NEW] A Reliable Transport Microservice is now available. The kdb Insights Reliable Transport (RT) is a microarchitecture for ensuring the reliable streaming of messages. It is designed to satisfy both high availability and high performance requirements.

Stream Processor

[NEW] Ingest large batches of data directly into the historical database. - Use fewer resources. Memory and disk resources greatly reduced during ingest. - New functionality available in the q and Python APIs.

Known Issues

Storage ManagerResource CoordinatorStream Processor

• If end-of-day processing is interrupted and remains incomplete until the next EOD period, this prevents further EODs from completing and requires manual recovery.

• Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

• Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

• If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

• If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

• If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

Upgrade considerations

Discovery Service

Spring Boot has been updated to version 3.0.0, this upgrade changes how profiles are activated within the application.yaml. If using a spring.profiles.active it should be updated to spring.config.active.on-profile. A working configuration is documented within the Service Discovery Microservice documentation

Full details on the Spring profiles change here

Artifacts

Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.

Service Discovery

microservice	Docker	Kubernetes
Discovery Registry	registry.dl.kx.com/kxi-eureka-discovery:1.4.0	https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.4.0.tgz
Discovery Proxy	registry.dl.kx.com/kxi-discovery-proxy:1.4.0

Stream Processor

microservice	Docker	qpk
Controller	registry.dl.kx.com/kxi-sp-controller:1.4.0
Worker	registry.dl.kx.com/kxi-sp-worker:1.4.0	https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.2.0/kxi-sp-worker.1.4.0.qpk
Coordinator	registry.dl.kx.com/kxi-sp-coordinator:1.4.0
Worker-Python	registry.dl.kx.com/kxi-sp-python:1.4.0
Worker-ML	registry.dl.kx.com/kxi-ml:1.4.0

Data Access

microservice	Docker
Data Access Process (DA)	registry.dl.kx.com/kxi-da:1.4.0
Single container	registry.dl.kx.com/kxi-da-single:1.4.0

Storage Manager

microservice	Docker
Storage Manager	registry.dl.kx.com/kxi-sm:1.4.0
End of Interval (EOI)	registry.dl.kx.com/kxi-sm-eoi:1.4.0
End of Day (EOD)	registry.dl.kx.com/kxi-sm-eod:1.4.0
Database Migration (DBM)	registry.dl.kx.com/kxi-sm-dbm:1.4.0
Single Container	registry.dl.kx.com/kxi-sm-single:1.4.0

Service Gateway

microservice	Docker
Gateway	registry.dl.kx.com/kxi-sg-gw:1.4.0
Resource Coordinator	registry.dl.kx.com/kxi-sg-rc:1.4.0
Aggregator	registry.dl.kx.com/kxi-sg-agg:1.4.0

User interface

microservice	Docker	Helm Chart
Dashboards	registry.dl.kx.com/kxi-gui-dash:1.4.0	https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.4.0.tgz

Machine Learning

microservice	package
ML Registry	https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-registry/0.8.0/kxi-ml-registry.0.8.0.qpk
ML Analytics	https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-analytics/0.8.0/kxi-ml-analytics.0.8.0.qpk

Reliable Transport

microservice	Docker	Kubernetes
Reliable Transport	registry.dl.kx.com/kxi-rt:1.4.0	https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-rt-1.4.0.tgz

Miscellaneous

microservice	Docker	package
Metrics Sidecar	registry.dl.kx.com/kxi-sidecar:1.4.0-rc1
OpenAPI SDK Generator	registry.dl.kx.com/kxi-openapi-codegen:0.8.0	https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar