Skip to content

Release Notes - Microservices 1.1.2

KX Insights Microservices 1.1.2 release.

Release Date

2022-07-15

Changes

Storage Manager & Data Access

[NEW] Late and out-of-order data handling

[NEW] Data received out of order is recorded with an appropriate ingest timestamp

[NEW] Late data is recorded, irrespective of where its correct destination is in the lifecycle (RDB, IDB, HDB)

[NEW] Late data is included by default for all requests, getData and ANSI SQL

[NEW] Late data can be filtered out (for example, to see only data present in the system at a particular point in time)

[NEW] Performance is not significantly impacted by late data arrival

[NEW] Reference data can be queried in the Explore Window, in a data pipeline and as part of the getData() API

[NEW] Support for non time-series reference data in Database

[NEW] Data in reference tables can be joined to time-series tables in free-form q queries or getData Improved SQL support for queries (joins and complex queries, except for reference tables)

[NEW] Compression support, including object storage

[NEW] Improved resiliency and performance

[FIX] The getData API endpoint no longer returns an error when an outputTZ is provided.

[FIX] Queries against the interval tier (idb) around the end-of-day rollover process could return zero results.

[FIX] Reference data joins which do not include the foreign column or where the foreign column has a different name are now possible.

[FIX] An issue where certain queries using count could throw the error Error: noupdate: `.Q `pn has been resolved.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below 
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

Upgrade considerations

Licensing

From 1.1.0 licensing changes are included to enable feature and comsumption based licensing in some cases. As part of this Microservices now require clients to upgrade to new licenses with feature flags. Previous licenses only required KXCE in the banner.

To validate if a license has feature flags, start a process or Microservice using it and check .z.l 4 in the terminal. This should return something like "insights.lib.embedq insights.lib.pykx insights.lib.sql insights.app.rt insig..

Database (Service Gateway, Data Access, Storage Manager)

  • Critical invalid configuration will now fatally exit the process with a FATAL log message and information on the configuration error
  • Assembly mounts.*.uri is now defined as mounts.*.baseURI - existing assembly files will need to be updated accordingly
  • Update to RT Client Library for new API signatures: https://code.kx.com/insights/1.1/microservices/storage-manager/sm-interface.html#interface-with-a-messaging-service

Artifacts

Service Discovery

microservice Docker Helm Chart
Discovery Registry registry.dl.kx.com/kxi-eureka-discovery:1.1.2 https://nexus.dl.kx.com/repository/kx-insights-charts/kxi-discovery-service-1.1.2.tgz
Discovery Proxy registry.dl.kx.com/kxi-discovery-proxy:1.1.0-rc.3

Stream Processor

microservice Docker qpk
Controller registry.dl.kx.com/kxi-sp-controller:1.1.1
Worker registry.dl.kx.com/kxi-sp-worker:1.1.1 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/1.1.1/kxi-sp-worker.1.1.1.qpk
Coordinator registry.dl.kx.com/kxi-sp-coordinator:1.1.1
Worker-Python registry.dl.kx.com/kxi-sp-python:1.1.1
Worker-ML registry.dl.kx.com/kxi-ml:1.1.1

Data Access

microservice Docker
Data Access Process (DA) registry.dl.kx.com/kxi-da:1.1.2

Storage Manager

microservice Docker
Storage Manager registry.dl.kx.com/kxi-sm:1.1.0
End of Interval (EOI) registry.dl.kx.com/kxi-sm-eoi:1.1.0
End of Day (EOD) registry.dl.kx.com/kxi-sm-eod:1.1.0
Database Migration (DBM) registry.dl.kx.com/kxi-sm-dbm:1.1.0
Single Container registry.dl.kx.com/kxi-sm-single:1.1.0

Service Gateway

microservice Docker
Gateway registry.dl.kx.com/kxi-sg-gw:1.1.0
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:1.1.2
Aggregator registry.dl.kx.com/kxi-sg-agg:1.1.1

User interface

microservice Docker Helm Chart
Dashboards registry.dl.kx.com/kxi-gui-dash:0.9.0 https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-0.9.0.tgz
Gateway registry.dl.kx.com/kxi-gui-gateway:1.0.0 https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-1.0.0.tgz
Data registry.dl.kx.com/kxi-gui-data:0.9.0 https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-0.9.0.tgz

Machine Learning

microservice package
ML Registry https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-registry/0.8.0/kxi-ml-registry.0.8.0.qpk
ML Analytics https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-analytics/0.8.0/kxi-ml-analytics.0.8.0.qpk

Miscellaneous

microservice Docker package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:1.1.7
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar