kdb Insights Microservices - Previous Release Notes
This page outlines release notes from past releases. For the most recent release, refer to the latest release page.
1.4.4
kdb Insights Microservices 1.4.4 release.
Release Date
2023-04-26
New Features
kdb Insights Database
[IMPROVEMENT]: SQL queries that filter by time but do not explicitly select the time column are now supported.
Fixes
Stream Processor
[FIX]: Window nodes in the Stream Processor can now accept dictionaries as input batches. Previously, incoming dictionaries would be merged with the input batch in the buffer, instead of appending to a buffered list of dictionaries.
Artifacts
Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.
Service Discovery
microservice | Docker | Kubernetes |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.4.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.4.1.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.4.1 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.4.5 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.4.5 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.2.0/kxi-sp-worker.1.4.0.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.4.5 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.4.5 | |
Worker-ML | registry.dl.kx.com/kxi-ml:1.4.5 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.4.5 |
Single container | registry.dl.kx.com/kxi-da-single:1.4.5 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.4.5 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.4.5 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.4.5 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.4.5 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.4.5 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.4.3 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.4.3 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.4.3 |
User Interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.4.3 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.4.3.tgz |
Machine Learning
Reliable Transport
microservice | Docker | Kubernetes |
---|---|---|
Reliable Transport | registry.dl.kx.com/kxi-rt:1.4.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-rt-1.4.0.tgz |
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.4.0-rc.1 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
Known Issues
- Users should disable
kurl
auto register within the kxi-sidecar container. Multiple registrations from component and sidecar could cause throttling.
Setting the below environment variable within the container spec will disable the auto register.
```yaml env: - name: KX_KURL_DISABLE_AUTO_REGISTER value: "1"
1.4.3
kdb Insights Microservices 1.4.3 release.
Release Date
2023-04-20
New Features
kdb Insights Database
[IMPROVEMENT]: The kdb Insights Database now supports a REST interface for loading historical datasets into an existing database. You can now perform a historical batch ingest using a simple REST API. For more details, see batch ingestion.
Fixes
Stream Processor
[FIX]: Security vulnerabilities patched in the Stream Processor for the following CVEs: CVE-2022-41910, CVE-2022-41902, CVE-2022-41900, CVE-2022-41880
kdb Insights Database
[FIX]: Migrating tiers to object storage could fail with the eventual error SMDBM Migration discontinued due to failure
if multiple threads were used.
[FIX]: If an SM client cannot connect to SM after a number of retries, it will now restart the pod to try to automatically recover. This resolves connection issues in DAPs when there are network connectivity issues.
[FIX]: The Service Gateway was creating duplicate connections to Resource Coordinators while they were waiting to come online. This was resulting in a large number of redundant connections being opened between the services. In large deployments, this could result in connection errors in the Resource Coordinator.
Artifacts
Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.
Service Discovery
microservice | Docker | Kubernetes |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.4.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.4.1.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.4.1 |
Stream Processor
microservice | Docker |
---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.4.4 |
Worker | registry.dl.kx.com/kxi-sp-worker:1.4.4 |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.4.4 |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.4.4 |
Worker-ML | registry.dl.kx.com/kxi-ml:1.4.4 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.4.4 |
Single container | registry.dl.kx.com/kxi-da-single:1.4.4 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.4.4 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.4.4 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.4.4 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.4.4 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.4.4 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.4.4 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.4.4 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.4.4 |
User interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.4.3 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.4.3.tgz |
Machine Learning
Reliable Transport
microservice | Docker | Kubernetes |
---|---|---|
Reliable Transport | registry.dl.kx.com/kxi-rt:1.4.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-rt-1.4.0.tgz |
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.4.4 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
Known Issues
- Users should disable
kurl
auto register within the kxi-sidecar container. Multiple registrations from component and sidecar could cause throttling.
Setting the below environment variable within the container spec will disable the auto register.
```yaml env: - name: KX_KURL_DISABLE_AUTO_REGISTER value: "1"
1.4.2
kdb Insights Microservices 1.4.2 release.
Release Date
2023-04-04
Fixes
Service Gateway
[FIX]: Fixed remote code injection vulnerability in Snake YAML CVE-2022-1471.
Stream Processor
[FIX]: Stream Processor file reader nodes (Azure, S3, File and Google storage) when attempting to read multiple files would only process the first file presented.
[FIX]: Stream Processor Azure file reader was failing when attempting to read a file with chunking enabled, resulting in limitations on the files that could be read.
[FIX]: Heartbeat logs were previously logged as INFO logs, they are now set to TRACE to reduce log output.
Storage Manager
[FIX]: Requests to object storage (Azure Storage, AWS S3, Google Cloud Storage) will now backoff and retry if they get a throttle request from the vendor. Jitter will be introduced to the retry to ensure that replicated processes do not all retry at the same time.
Artifacts
Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.
Service Discovery
microservice | Docker | Kubernetes |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.4.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.4.1.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.4.1 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.4.3 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.4.3 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.2.0/kxi-sp-worker.1.4.0.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.4.3 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.4.3 | |
Worker-ML | registry.dl.kx.com/kxi-ml:1.4.3 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.4.3 |
Single container | registry.dl.kx.com/kxi-da-single:1.4.3 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.4.3 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.4.3 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.4.3 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.4.3 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.4.3 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.4.3 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.4.3 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.4.3 |
User interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.4.3 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.4.3.tgz |
Machine Learning
Reliable Transport
microservice | Docker | Kubernetes |
---|---|---|
Reliable Transport | registry.dl.kx.com/kxi-rt:1.4.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-rt-1.4.0.tgz |
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.4.0-rc.1 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
Known Issues
- Users should disable
kurl
auto register within the kxi-sidecar container. Multiple registrations from component and sidecar could cause throttling.
Setting the below environment variable within the container spec will disable the auto register.
env:
- name: KX_KURL_DISABLE_AUTO_REGISTER
value: "1"
1.4.1
Release Date
2023-03-14
Fixes
Storage Manager
- If end-of-day processing is interrupted and remains incomplete until the next EOD period, this prevents further EODs from completing and requires manual recovery. This issue has been resolved.
Artifacts
Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.
Service Discovery
microservice | Docker | Kubernetes |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.4.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.4.0.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.4.0 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.4.0 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.4.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.2.0/kxi-sp-worker.1.4.0.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.4.0 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.4.0 | |
Worker-ML registry.dl.kx.com/kxi-ml:1.4.0 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.4.0 |
Single container | registry.dl.kx.com/kxi-da-single:1.4.0 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.4.1 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.4.1 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.4.1 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.4.1 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.4.1 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.4.0 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.4.0 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.4.0 |
User interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.4.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.4.0.tgz |
Machine Learning
Reliable Transport
microservice | Docker | Kubernetes |
---|---|---|
Reliable Transport | registry.dl.kx.com/kxi-rt:1.4.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-rt-1.4.0.tgz |
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.4.0-rc.1 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.4.0
kdb Insights Microservices 1.4.0 release.
Release Date
2023-01-31
New Features
Reliable Transport
A Reliable Transport Microservice is now available.
Changes
Known Issues
Upgrade considerations
Discovery Service
Spring Boot has been updated to version 3.0.0, this upgrade changes how profiles are activated within the application.yaml. If using a spring.profiles.active
it should be updated to spring.config.active.on-profile
. A working configuration is documented within the Service Discovery Microservice documentation.
Full details on the Spring profiles change here
Artifacts
Service Discovery
microservice | Docker | Kubernetes |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.4.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.4.0.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.4.0 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.2.0 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.2.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.2.0/kxi-sp-worker.1.2.0.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.2.0 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.2.0 | |
Worker-ML | registry.dl.kx.com/kxi-ml:1.2.0 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.2.0 |
Single container | registry.dl.kx.com/kxi-da-single:1.2.0 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.2.0 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.2.0 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.2.0 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.2.0 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.2.0 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.2.0 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.2.0 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.2.0 |
User interface
microservice | Docker | Kubernetes |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.1.2 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.2.0.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:1.1.2 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-1.2.0.tgz |
Data | registry.dl.kx.com/kxi-gui-data:1.1.2 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-1.2.0.tgz |
Machine Learning
Reliable Transport
microservice | Docker | Kubernetes |
---|---|---|
RT |
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.4.0 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.3.6
kdb Insights 1.3.6 release.
Release Date
2023-04-06
Fixes
Storage Manager
- Migrating tiers to object storage could fail with the eventual error
SMDBM Migration discontinued due to failure
if multiple threads were used. This has been resolved.
Artifacts
Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.
Service Discovery
microservice | Docker | Helm Chart |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.3.1.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.3.1 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.3.1 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.3.1/kxi-sp-worker.1.3.1.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.3.1 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.3.1 | |
Worker-ML | registry.dl.kx.com/kxi-ml:1.3.1 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.3.9 |
Single container | registry.dl.kx.com/kxi-da-single:1.3.9 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.3.10 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.3.10 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.3.10 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.3.10 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.3.10 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.3.3 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.3.3 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.3.3 |
User interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.3.0.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-1.3.0.tgz |
Data | registry.dl.kx.com/kxi-gui-data:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-1.3.0.tgz |
Machine Learning
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.3.0 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.3.5
kdb Insights 1.3.5 release.
Release Date
2023-03-16
Fixes
Storage Manager and Data Access
- Storage Manager and Data Access processes now retry, with exponential backoff, when object storage authentication fails. Previously, rate limiting of authentication token refreshes on large kdb Insights deployments could cause access failure, and subsequent processing issues. For more detail, see the release notes for Core 4.0.2
Artifacts
Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.
Service Discovery
microservice | Docker | Helm Chart |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.3.1.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.3.1 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.3.1 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.3.1/kxi-sp-worker.1.3.1.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.3.1 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.3.1 | |
Worker-ML | registry.dl.kx.com/kxi-ml:1.3.1 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.3.9 |
Single container | registry.dl.kx.com/kxi-da-single:1.3.9 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.3.9 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.3.9 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.3.9 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.3.9 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.3.9 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.3.3 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.3.3 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.3.3 |
User interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.3.0.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-1.3.0.tgz |
Data | registry.dl.kx.com/kxi-gui-data:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-1.3.0.tgz |
Machine Learning
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.3.0 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.3.3
kdb Insights 1.3.3 release.
Release Date
2023-03-07
Fixes
Storage Manager
- If end-of-day processing is interrupted and remains incomplete until the next EOD period, this prevents further EODs from completing and requires manual recovery. This issue has been resolved.
Artifacts
Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.
Service Discovery
microservice | Docker | Helm Chart |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.3.1.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.3.1 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.3.1 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.3.1/kxi-sp-worker.1.3.1.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.3.1 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.3.1 | |
Worker-ML | registry.dl.kx.com/kxi-ml:1.3.1 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.3.3 |
Single container | registry.dl.kx.com/kxi-da-single:1.3.3 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.3.8 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.3.8 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.3.8 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.3.8 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.3.8 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.3.3 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.3.3 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.3.3 |
User interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.3.0.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-1.3.0.tgz |
Data | registry.dl.kx.com/kxi-gui-data:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-1.3.0.tgz |
Machine Learning
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.3.0 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.3.3
kdb Insights 1.3.3 release.
Release Date
2023-03-01
Fixes
Storage Manager
- A limitation to the number of objects that could be present in an object storage tier has been fixed. This could cause EOD data migration to object storage tiers to fail in SM version 1.3.6.
Artifacts
Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.
Service Discovery
microservice | Docker | Helm Chart |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.3.1.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.3.1 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.3.1 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.3.1/kxi-sp-worker.1.3.1.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.3.1 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.3.1 | |
Worker-ML | registry.dl.kx.com/kxi-ml:1.3.1 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.3.3 |
Single container | registry.dl.kx.com/kxi-da-single:1.3.3 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.3.7 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.3.7 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.3.7 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.3.7 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.3.7 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.3.3 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.3.3 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.3.3 |
User interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.3.0.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-1.3.0.tgz |
Data | registry.dl.kx.com/kxi-gui-data:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-1.3.0.tgz |
Machine Learning
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.3.0 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.3.2
kdb Insights 1.3.2 release.
Release Date
2023-02-21
Fixes
Storage Manager
-
Object storage copy limits (2GB) no longer block SM from migrating large columns to an object storage tier.
-
Column additions to splayed tables could sometimes fail when attempting to update the schema. This has been fixed.
Artifacts
Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.
Service Discovery
microservice | Docker | Helm Chart |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.3.1.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.3.1 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.3.1 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.3.1/kxi-sp-worker.1.3.1.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.3.1 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.3.1 | |
Worker-ML | registry.dl.kx.com/kxi-ml:1.3.1 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.3.3 |
Single container | registry.dl.kx.com/kxi-da-single:1.3.3 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.3.6 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.3.6 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.3.6 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.3.6 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.3.6 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.3.3 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.3.3 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.3.3 |
User interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.3.0.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-1.3.0.tgz |
Data | registry.dl.kx.com/kxi-gui-data:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-1.3.0.tgz |
Machine Learning
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.3.0 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.3.1
kdb Insights Microservices 1.3.1 release.
Release Date
2022-12-06
New Features
Storage Manager
- SM now supports writing its own inventory file, under the tier settings. Inventory files can speed load times for databases - you can read more here.
The DA should be configured to set
- name: s3 mount: hb store: s3://kxi-sm-test/bucketroot/db inventory: enabled: true location: s3://kxi-sm-test/bucketroot/inventory.tgz
KX_OBJSTR_INVENTORY_FILE
as an env var to a path relative to the root of the bucket. For examplebucketroot/inventory.tgz
.
DA/Aggregator
- The garbage collection in the DAP and Aggregator processes has been moved to a staggered timer for improved query throughput. The default is 10 minutes, and is configurable through the
KXI_GC_FREQ
environment variables on each service. Previously, the garbage collection ran after each query.
Fixes
Storage Manager
-
Tables with names starting with underscores are no longer considered invalid.
-
Schema changes no longer fail on assemblies with configured—but empty—object storage tiers.
Data Access
- An upgrade issue where DAs could register with SM before all tables were ready has been fixed. This could result in
Error executing listener for event discovery
Artifacts
Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.
Service Discovery
microservice | Docker | Helm Chart |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.3.1.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.3.1 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.3.1 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.3.1/kxi-sp-worker.1.3.1.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.3.1 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.3.1 | |
Worker-ML | registry.dl.kx.com/kxi-ml:1.3.1 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.3.3 |
Single container | registry.dl.kx.com/kxi-da-single:1.3.3 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.3.3 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.3.3 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.3.3 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.3.3 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.3.3 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.3.3 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.3.3 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.3.3 |
User interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.3.0.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-1.3.0.tgz |
Data | registry.dl.kx.com/kxi-gui-data:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-1.3.0.tgz |
Machine Learning
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.3.0 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.3.0
kdb Insights Microservices 1.3.0 release.
Release Date
2022-11-22
Storage Manager
[FIX] Multiple issues due to connections not being re-established if they dropped and tty being enabled in some scenarios causing debug prompts.
[FIX] An issue where the storage manager exhausted all available connections when registering DA clients that were repeatedly failing.
Service Gateway
[FIX] A problem where the gateway would not respond to a query if it died during query processing due to an out of memory error. The gateway will now issue timeouts for queries that have exceeded their original timeout by more than 5 seconds.
Metrics
[FIX] An issue where enabling metrics for the .z.ps handler caused the 0
handle to not return results.
Known Issues
Artifacts
Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.
Service Discovery
microservice | Docker | Helm Chart |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.3.1.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.3.1 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.3.1 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.3.1/kxi-sp-worker.1.3.1.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.3.1 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.3.1 | |
Worker-ML | registry.dl.kx.com/kxi-ml:1.3.1 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.3.1 |
Single container | registry.dl.kx.com/kxi-da-single:1.3.1 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.3.1 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.3.1 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.3.1 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.3.1 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.3.1 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.3.1 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.3.1 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.3.1 |
User interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.3.0.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-1.3.0.tgz |
Data | registry.dl.kx.com/kxi-gui-data:1.3.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-1.3.0.tgz |
Machine Learning
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.3.0 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.2.1
kdb Insights Microservices 1.2.1 release.
Release Date
2022-10-20
Storage Manager
[FIX] An upgrade issue affecting migrations from kdb Insights Microservices 1.1 using object storage tiers has been fixed.
[FIX] A problem where underlying system calls could cause Storage Manager to hang during EOI or EOD has been resolved.
[FIX] Discovery issues with DAPs when Storage Manager was used in sm-single mode have been resolved.
[FIX] Storage Manager now retries if it's unable to initially reach the Reliable Transport.
Licensing
[FIX] Airgap licensing problems due to lack of support for regional AWS STS endpoints have been resolved.
Known Issues
- If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below:
{"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
.qsp.read.fromDatabase
cannot be run in a pipeline to query a kdb Insights database. The following example code can be used as a workaround to query data (ex. reference data to be used within a pipeline).
onResponse:{[hdr;data]
if[0 <> hdr`ac;
.sp.log.error ("Query failed host=%s, api=%s, code=%n, error=%s";hdr`gw; hdr`api; hdr`ac; hdr`ai)];
publish data
}
.qsp.onStart {
h:hopen hsym `$"sg-gateway-host:5050";
res: neg[h] (`.kxi.getData; (enlist `table)!(enlist `instruments); `onResponse; (0#`)!());
}
.qsp.run .qsp.read.fromCallback[`publish] .qsp.write.toConsole[]
- If metrics reporting is enabled within a microservice, along with the handler for
.z.ps
, services calling through the.z.ps
with0(f;x)
will throw atype
error due to a failure in the handler to return correctly. The workaround, if metrics reporting is enabled, is to disable the.z.ps
event handler in the metrics configuration for all microservices.
"metrics": {
"enabled": "true",
"frequency": 5,
"handler": {
"pc": true,
"pg": true,
"ph": true,
"po": true,
"pp": true,
"ps": false,
"ts": true,
"wc": true,
"wo": true,
"ws": true
}
}
Upgrade considerations
- If using a
kdb+ tick
tickerplant, the schema for database signals has changed to the following, which must be updated in the ticksym.q
(`$"_prtnEnd") set ([] time:"n"$(); sym:`$(); signal:`$(); endTS:"p"$(); opts:())
(`$"_reload") set ([] time:"n"$(); sym:`$(); mount:`$(); params:(); asm:`$())
Artifacts
Highlights denote artifacts which have been updated as part of this patch release. All other artifacts are unchanged from the previous major or minor release.
Service Discovery
microservice | Docker | Helm Chart |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.2.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.2.1.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.2.0 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.2.0 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.2.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.2.0/kxi-sp-worker.1.2.0.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.2.0 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.2.0 | |
Worker-ML | registry.dl.kx.com/kxi-ml:1.2.0 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.2.1 |
Single container | registry.dl.kx.com/kxi-da-single:1.2.1 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.2.1 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.2.1 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.2.1 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.2.1 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.2.1 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.2.1 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.2.1 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.2.1 |
User interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.1.2 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.2.0.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:1.1.2 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-1.2.0.tgz |
Data | registry.dl.kx.com/kxi-gui-data:1.1.2 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-1.2.0.tgz |
Machine Learning
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.2.0 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.2.0
KX Insights Microservices 1.2.0 release.
Release Date
2022-09-02
Changes
Storage Manager & Data Access
[NEW] Ability for Data Access tiers (RDB, IDB, HDB) to share compute resources by using mountList
within Assembly configuration rather than mountName
.
Stream Processor
[NEW] MQTT QoS 0 Reader has been added.
Known Issues
- If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
{"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
.qsp.read.fromDatabase
cannot be run in a pipeline to query an Insights database. The following example code can be used as a workaround to query data (ex. reference data to be used within a pipeline).
onResponse:{[hdr;data]
if[0 <> hdr`ac;
.sp.log.error ("Query failed host=%s, api=%s, code=%n, error=%s";hdr`gw; hdr`api; hdr`ac; hdr`ai)];
publish data
}
.qsp.onStart {
h:hopen hsym `$.spenv.getReleaseName[],"-sg-gateway:5050";
res: neg[h] (`.kxi.getData; (enlist `table)!(enlist `instruments); `onResponse; ()!());
}
.qsp.run .qsp.read.fromCallback[`publish] .qsp.write.toConsole[]
- If metrics reporting is enabled within a microservice, along with the handler for
.z.ps
, services calling through the.z.ps
with0(f;x)
will throw atype
error due to a failure in the handler to return correctly. The workaround if metrics reporting is enabled is to disable the.z.ps
event handler in the metrics configuration for all microservices.
"metrics": {
"enabled": "true",
"frequency": 5,
"handler": {
"pc": true,
"pg": true,
"ph": true,
"po": true,
"pp": true,
"ps": false,
"ts": true,
"wc": true,
"wo": true,
"ws": true
}
}
Upgrade considerations
- If using a
kdb+ tick
tickerplant, the schema for database signals has changed to the following, which must be updated in the ticksym.q
(`$"_prtnEnd") set ([] time:"n"$(); sym:`$(); signal:`$(); endTS:"p"$(); opts:());
(`$"_reload") set ([] time:"n"$(); sym:`$(); mount:`$(); params:(); asm:`$())
Artifacts
Service Discovery
microservice | Docker | Helm Chart |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.2.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.2.0.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.2.0 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.2.0 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.2.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.2.0/kxi-sp-worker.1.2.0.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.2.0 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.2.0 | |
Worker-ML | registry.dl.kx.com/kxi-ml:1.2.0 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.2.0 |
Single container | registry.dl.kx.com/kxi-da-single:1.2.0 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.2.0 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.2.0 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.2.0 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.2.0 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.2.0 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.2.0 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.2.0 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.2.0 |
User interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.1.2 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.2.0.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:1.1.2 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-1.2.0.tgz |
Data | registry.dl.kx.com/kxi-gui-data:1.1.2 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-1.2.0.tgz |
Machine Learning
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.2.0 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.1.3
KX Insights Microservices 1.1.3 release.
Release Date
2022-08-17
Changes
Storage Manager & Data Access
[FIX] Fix for Alpine vulnerability CVE-2022-37434
Stream Processor
[FIX] Fix for Alpine vulnerability CVE-2022-37434
Service Discovery
[FIX] Fix for Alpine vulnerabilities CVE-2022-37434 and CVE-2022-32207
Sidecar
[FIX] Fix for Alpine vulnerabilities CVE-2022-37434
Known Issues
-
If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below:
{"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
- If metrics reporting is enabled within a microservice, along with the handler for
.z.ps
, services calling through the.z.ps
with0(f;x)
will throw atype
error due to a failure in the handler to return correctly. The workaround is if metrics reporting is enabled, disable the.z.ps
event handler in the metrics configuration for all microservices.
"metrics": {
"enabled": "true",
"frequency": 5,
"handler": {
"pc": true,
"pg": true,
"ph": true,
"po": true,
"pp": true,
"ps": false,
"ts": true,
"wc": true,
"wo": true,
"ws": true
}
}
Upgrade considerations
Licensing
From 1.1.0 licensing changes are included to enable feature and comsumption based licensing in some cases. As part of this Microservices now require clients to upgrade to new licenses with feature flags. Previous licenses only required KXCE
in the banner.
To validate if a license has feature flags, start a process or Microservice using it and check .z.l 4
in the terminal. This should return something like "insights.lib.embedq insights.lib.pykx insights.lib.sql insights.app.rt insig..
Database (Service Gateway, Data Access, Storage Manager)
- Critical invalid configuration will now fatally exit the process with a
FATAL
log message and information on the configuration error - Assembly
mounts.*.uri
is now defined asmounts.*.baseURI
- existing assembly files will need to be updated accordingly - Update to RT Client Library for new API signatures: https://code.kx.com/insights/1.1/microservices/storage-manager/sm-interface.html#interface-with-a-messaging-service
Artifacts
Service Discovery
microservice | Docker | Helm Chart |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.1.3 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.1.4.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.1.3 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.1.3 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.1.3 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.1.3/kxi-sp-worker.1.1.3.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.1.3 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.1.3 | |
Worker-ML | registry.dl.kx.com/kxi-ml:1.1.3 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.1.2 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.1.0 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.1.0 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.1.0 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.1.0 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.1.1 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.1.2 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.1.1 |
User interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:1.1.2 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-1.1.2.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:1.1.2 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-1.1.2.tgz |
Data | registry.dl.kx.com/kxi-gui-data:1.1.2 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-1.1.2.tgz |
Machine Learning
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.1.8 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.1.2
Release Date
2022-07-15
Changes
Storage Manager & Data Access
[NEW] Late and out-of-order data handling
[NEW] Data received out of order is recorded with an appropriate ingest timestamp
[NEW] Late data is recorded, irrespective of where its correct destination is in the lifecycle (RDB, IDB, HDB)
[NEW] Late data is included by default for all requests, getData
and ANSI SQL
[NEW] Late data can be filtered out (for example, to see only data present in the system at a particular point in time)
[NEW] Performance is not significantly impacted by late data arrival
[NEW] Reference data can be queried in the Explore Window, in a data pipeline and as part of the getData()
API
[NEW] Support for non time-series reference data in Database
[NEW] Data in reference tables can be joined to time-series tables in free-form q queries or getData
Improved SQL support for queries (joins and complex queries, except for reference tables)
[NEW] Compression support, including object storage
[NEW] Improved resiliency and performance
[FIX] The getData API endpoint no longer returns an error when an outputTZ
is provided.
[FIX] Queries against the interval tier (idb) around the end-of-day rollover process could return zero results.
[FIX] Reference data joins which do not include the foreign column or where the foreign column has a different name are now possible.
[FIX] An issue where certain queries using count
could throw the error Error: noupdate: `.Q `pn
has been resolved.
Known Issues
- If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
{"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
Upgrade considerations
Licensing
From 1.1.0 licensing changes are included to enable feature and comsumption based licensing in some cases. As part of this Microservices now require clients to upgrade to new licenses with feature flags. Previous licenses only required KXCE
in the banner.
To validate if a license has feature flags, start a process or Microservice using it and check .z.l 4
in the terminal. This should return something like "insights.lib.embedq insights.lib.pykx insights.lib.sql insights.app.rt insig..
Database (Service Gateway, Data Access, Storage Manager)
- Critical invalid configuration will now fatally exit the process with a
FATAL
log message and information on the configuration error - Assembly
mounts.*.uri
is now defined asmounts.*.baseURI
- existing assembly files will need to be updated accordingly - Update to RT Client Library for new API signatures: https://code.kx.com/insights/1.1/microservices/storage-manager/sm-interface.html#interface-with-a-messaging-service
Artifacts
Service Discovery
microservice | Docker | Helm Chart |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.1.2 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.1.2.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:1.1.0-rc.3 |
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.1.1 | |
Worker | registry.dl.kx.com/kxi-sp-worker:1.1.1 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.1.1/kxi-sp-worker.1.1.1.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.1.1 | |
Worker-Python | registry.dl.kx.com/kxi-sp-python:1.1.1 | |
Worker-ML | registry.dl.kx.com/kxi-ml:1.1.1 |
Data Access
microservice | Docker |
---|---|
Data Access Process (DA) | registry.dl.kx.com/kxi-da:1.1.2 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.1.0 |
End of Interval (EOI) | registry.dl.kx.com/kxi-sm-eoi:1.1.0 |
End of Day (EOD) | registry.dl.kx.com/kxi-sm-eod:1.1.0 |
Database Migration (DBM) | registry.dl.kx.com/kxi-sm-dbm:1.1.0 |
Single Container | registry.dl.kx.com/kxi-sm-single:1.1.0 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.1.0 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.1.2 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.1.1 |
User interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-0.9.0.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:1.0.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-1.0.0.tgz |
Data | registry.dl.kx.com/kxi-gui-data:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-0.9.0.tgz |
Machine Learning
microservice | package |
---|---|
ML Registry | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-registry/0.8.0/kxi-ml-registry.0.8.0.qpk |
ML Analytics | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-analytics/0.8.0/kxi-ml-analytics.0.8.0.qpk |
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:1.1.7 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
1.1.0
KX Insights Microservices 1.1.0 release.
Release Date
2022.03.14
Stream Processor
Microservice | Docker | Helm Chart |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:1.0.1 | kxi-sp-1.0.0.tgz |
Worker | registry.dl.kx.com/kxi-sp-worker:1.0.1 | |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:1.0.1 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:1.0.2 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:1.0.2 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:1.0.2 |
Data Access
microservice | Docker |
---|---|
Data Access Process(DA) | registry.dl.kx.com/kxi-da:1.0.2 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:1.0.3 |
End of Interval(EOI) | registry.dl.kx.com/kxi-sm-eoi:1.0.3 |
End of Day(EOD) | registry.dl.kx.com/kxi-sm-eod:1.0.3 |
Database Migration(DBM) | registry.dl.kx.com/kxi-sm-dbm:1.0.3 |
User Interface
Microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:0.9.0-1 | gui-dash-1.0.0.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:1.0.0 | |
Data | registry.dl.kx.com/kxi-gui-data:0.10.4 |
0.11.0
Beta Release Patch
Release Date
2021.12.14
Service Discovery
[FIX] Upgrade from version 2.13 to 2.15 of log4j2 to fix CVE
Microservice | Docker | Helm Chart |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.0.5 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.0.3.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:0.9.3 |
0.10.0
Beta Release Update
Release Date
2022.12.08
Service Discovery
Microservice | Docker |
---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.0.4 |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:0.9.1 |
Stream Processor
microservice | Docker |
---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:0.11.4 |
Worker | registry.dl.kx.com/kxi-sp-worker:0.11.4 |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:0.11.4 |
Service Gateway
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:0.9.37 |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:0.9.36 |
Aggregator | registry.dl.kx.com/kxi-sg-agg:0.9.36 |
Data Access
microservice | Docker |
---|---|
Data Access Process(DA) | registry.dl.kx.com/kxi-da:0.9.33 |
Storage Manager
microservice | Docker |
---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:0.9.16 |
End of Interval(EOI) | registry.dl.kx.com/kxi-sm-eoi:0.9.16 |
End of Day(EOD) | registry.dl.kx.com/kxi-sm-eod:0.9.16 |
Database Migration(DBM) | registry.dl.kx.com/kxi-sm-dbm:0.9.16 |
User Interface
microservice | Docker |
---|---|
Gateway | registry.dl.kx.com/kxi-gui-gateway:0.10.0 |
Data | registry.dl.kx.com/kxi-gui-data:0.10.0 |
0.9.0
Beta Update
Release Date
2021.08.18
Stream Processor
microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:0.9.0 | |
Worker | registry.dl.kx.com/kxi-sp-worker:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/0.9.0/kxi-sp-worker.0.9.0.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-coordinator:0.9.0 |
[NEW] Stream Processor now supports arbitrarily nested namespaces to include operators
[NEW] API added to allow extract of metrics and status for Stream Processor pipelines
[NEW] Pipeline recovery from Controller fault. Workers die after configurable timeout rather than right away and workers retry connection to Controller host:port periodically after disconnect
[NEW] Support added for chunked file reading. Chunked reading will leverage a concept called 'partials'. Readers will indicate if they write 'partials' into the stream. This will be identified by a new optional emit partials discovery function in the reader's plugin interface. By default this function will return false. If a reader emits partials, the next operator in the pipeline can indicate it buffers partials. If a plugin can buffer partials the partials will be sent directly to the operator. If it cannot read partials, a buffer operator will automatically be inserted in the pipeline that will buffer all partials until a full message is available. If an operator emits partials, it must implement a combine function that merges partial results. This combiner will be made available to the downstream operator.
Proposed API additions:
generatesPartials - This operator will output partial results emitsPartials - Should partial results be passed downstream if the next operator can handle it? combinePartials - Given n sets of partials, combine the data processesPartials - Indicates an operator can process partial data usePartials - Indicates to the operator that data will be provided as partials
The result will allow readers to emit partial results. If decoders can process partials directly they will be given the partials function and partials when available. If they cannot, data will be buffered until a full message is emitted. This will also allow reduce operations to be distributed with partial results being emitted to a combine stage and being buffered before they are finally emitted.
[NEW] API for retrieving the DAG spec from a running pipeline to manifest in the UI
[NEW] Pipeline teardown API - REST clients such as the UI should be able to request the teardown of a pipeline to the coordinator. Teardown should request a teardown from all workers. After the workers cleanly exit, the controller and worker resources need to be deallocated. This includes removing pods, stateful sets, services, etc.
[NEW] Coordinators now use heartbeats to identify pipelines that are still active
[NEW] Kubernetes workloads to StatefulSets, for future integration with RT as well as better persistence of checkpointing
[NEW] Kafka reader 'fromBeginning' / 'fromEnd' usability added
[NEW] Length error running passing some atoms to a pipeline
.qsp.run
.qsp.read.fromCallback[`publish; `publish]
.qsp.map[`myMap; {x+1}]
.qsp.write.toConsole[]
If you pass this pipeline the atom 1, it works fine
publish 1 But if you call it on something with more than one character, you get a length error in .spwrite.console.write
publish 100 I'm not sure what the error is here. Can pipelines accept atoms, or is the issue that .sp.console.write isn't handling them? If pipelines can't accept atoms, it should error earlier
[NEW] Stream Processor now includes CSV Decoder - operators and basic type casting
[FIX] .sp.push isn't working for operators with foreign values as parameters
[FIX] Kafka commit offset on checkpoint
[FIX] Kafka library was not reading from the start of a partition
[FIX] .qsp.run was not starting all parent pipelines
[FIX] Added cleanup callback on pipeline teardown
[FIX] .qsp.filter should verify the type of the return value
[FIX] Runtime fails to compile when operations have the same ID
[FIX] isAsync dict is was incorrectly formed if two plugins had the same ID
Data Access
microservice | Docker | qpk |
---|---|---|
Data Access Process(DA) | registry.dl.kx.com/kxi-da:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-da/0.9.0/kxi-da.0.9.0.qpk |
Data Access Operator | registry.dl.kx.com/kxi-da-operator:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-da-operator/0.9.0/kxi-da-operator.0.9.0.qpk |
[NEW] DA now return failure codes when a query is unsuccessful
[NEW] Definition and implementation of what metrics the DA microservice will expose
[NEW] React to data lifecycle events
[NEW] Determine, fulfill, and report temporal purview of process at start-up
Storage Manager
microservice | Docker | qpk |
---|---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sm/0.9.0/kxi-sm.0.9.0.qpk |
End of Interval(EOI) | registry.dl.kx.com/kxi-sm-eoi:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sm-eoi/0.9.0/kxi-sm-eoi.0.9.0.qpk |
End of Day(EOD) | registry.dl.kx.com/kxi-sm-eod:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sm-eod/0.9.0/kxi-sm-eod.0.9.0.qpk |
Database Migration(DBM) | registry.dl.kx.com/kxi-sm-dbm:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sm-dbm/0.9.0/kxi-sm-dbm.0.9.0.qpk |
[NEW] Support for pushing data to S3 via SM
[NEW] Added ObjStor tier option for initial import
[FIX] Fix wrong RDB and IDB purview for freshly imported DB
[FIX] Kurl updates for on-prem DBs
[NEW] Utility to copy data to object storage
[NEW] Updated partition layout options w.r.t. object storage
[NEW] Migration integration with EOD
[NEW] Add tier compression
[NEW] Generic/custom tickerplant as EMS substrate
Service Gateway
microservice | Docker | qpk |
---|---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:0.9.1 | |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:0.9.1 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sg-rc/0.9.1/kxi-sg-rc.0.9.1.qpk |
Aggregator | registry.dl.kx.com/kxi-sg-agg:0.9.1 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sg-agg/0.9.1/kxi-sg-agg.0.9.1.qpk |
[NEW] Surface assembly info and table schemas via SG's getMeta API
[NEW] Added SG asymmetric DAP mode
[NEW] Added Health and Readiness checks for Service Gateway
[NEW] Added request timeout feature - detection and failure.
[NEW] Metadata retrieval API for SG/DA
[NEW] Support for custom code loading in SG and DA
[FIX] Fix to reconnection logic for coordinator when connection is dropped
[NEW] Added Health and Readyness checks for Service Gateway
[NEW] Added request timeout feature - detection and failure.
[NEW] Metadata retrieval API for SG/DA
[NEW] Support for custom code loading in SG and DA
[FIX] Fix to reconnection logic for coordinator when conection is dropped
[FIX] Aggregator sidecar erroring when trying to register with discovery
[FIX] If discovery is up, but then getServices() fails, we did not retry
User Interface
microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-dash:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-0.9.0.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-0.9.0.tgz |
Data | registry.dl.kx.com/kxi-gui-data:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-0.9.0.tgz |
Machine Learning
microservice | package |
---|---|
ML Registry | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-registry/0.8.0/kxi-ml-registry.0.8.0.qpk |
ML Analytics | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-analytics/0.8.0/kxi-ml-analytics.0.8.0.qpk |
[NEW] ML Analytics - A centralized location for all machine-learning analytics produced by KX
[NEW] ML Registry - The ML Registry defines a centralized location for storing versioned machine-learning models and advanced analytics alongside parameters, metrics and other important artifacts data scientists to store information related to their machine-learning workflows.
Miscellaneous
microservice | Docker | package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:0.8.0 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |
0.8.0
Initial Beta Release for all microservices.
Release Date
2021.06.30
Service Discovery
[NEW] A cloud-native, scalable registry which stores, manages and distributes running services and their metadata for all other services within the Microservice driven application.
Microservice | Docker | Helm Chart |
---|---|---|
Discovery Registry | registry.dl.kx.com/kxi-eureka-discovery:1.0.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-0.8.0.tgz |
Discovery Proxy | registry.dl.kx.com/kxi-discovery-proxy:0.8.2 |
Stream Processor
[NEW] A high-performance, scalable event-stream processing system implemented in q.
Microservice | Docker | qpk |
---|---|---|
Controller | registry.dl.kx.com/kxi-sp-controller:0.9.0 | |
Worker | registry.dl.kx.com/kxi-sp-worker:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/0.9.0/kxi-sp-worker.0.9.0.qpk |
Coordinator | registry.dl.kx.com/kxi-sp-worker:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/0.9.0/kxi-sp-worker.0.9.0.qpk |
Data Access
[NEW] Provides read-only access to all data stored in a database, regardless of where the data currently resides in the data lifecycle.
Microservice | Docker | qpk |
---|---|---|
Data Access Process(DA) | registry.dl.kx.com/kxi-da:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-da/0.9.0/kxi-da.0.9.0.qpk |
Data Access Operator | registry.dl.kx.com/kxi-da-operator:0.9.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-da-operator/0.9.0/kxi-da-operator.0.9.0.qpk |
Storage Manager
[NEW] Handles data persistence and migration.
Microservice | Docker | qpk |
---|---|---|
Storage Manager | registry.dl.kx.com/kxi-sm:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sm/0.8.0/kxi-sm.0.8.0.qpk |
End of Interval(EOI) | registry.dl.kx.com/kxi-sm-eoi:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sm-eoi/0.8.0/kxi-sm-eoi.0.8.0.qpk |
End of Day(EOD) | registry.dl.kx.com/kxi-sm-eod:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sm-eod/0.8.0/kxi-sm-eod.0.8.0.qpk |
Database Migration(DBM) | registry.dl.kx.com/kxi-sm-dbm:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sm-dbm/0.8.0/kxi-sm-dbm.0.8.0.qpk |
Gateway Service
[NEW] A unified access point for data that is spread across multiple processes.
Microservice | Docker | qpk |
---|---|---|
Gateway | registry.dl.kx.com/kxi-sg-gw:0.9.1 | |
Resource Coordinator | registry.dl.kx.com/kxi-sg-rc:0.9.1 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sg-rc/0.9.1/kxi-sg-rc.0.9.1.qpk |
Aggregator | registry.dl.kx.com/kxi-sg-agg:0.9.1 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sg-agg/0.9.1/kxi-sg-agg.0.9.1.qpk |
REST Server | registry.dl.kx.com/kxi-sg-rest:0.9.1 | |
REST Proxy | registry.dl.kx.com/kxi-sg-rest-proxy:0.9.1 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sg-rest-proxy/0.9.1/kxi-sg-rest-proxy.0.9.1.qpk |
User Interface
[NEW] An interactive data visualization tool that enables non-technical and power users to query, transform, share and present live data insights.
Microservice | Docker | Helm Chart |
---|---|---|
Dashboards | registry.dl.kx.com/kxi-gui-direct:0.8.2 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-direct-0.8.2.tgz |
Gateway | registry.dl.kx.com/kxi-gui-gateway:0.8.1 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-0.8.1.tgz |
Data | registry.dl.kx.com/kxi-gui-data:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-0.8.0.tgz |
Miscellaneous
[NEW] Prometheus metrics sidecar image that provides four metric types: counter, gauge, histogram, summary.
[NEW] OpenAPI specifications for all microservices, and a library for generating server stubs from an OpenAPI specification that uses our REST server library.
Microservice | Docker | Package |
---|---|---|
Metrics Sidecar | registry.dl.kx.com/kxi-sidecar:0.8.0 | |
OpenAPI SDK Generator | registry.dl.kx.com/kxi-openapi-codegen:0.8.0 | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar |