Skip to content

Release Notes - Insights Services

Beta Update

Release Date

2021.08.18

Stream Processor

microservice Docker qpk
Controller registry.dl.kx.com/kxi-sp-controller:0.9.0
Worker registry.dl.kx.com/kxi-sp-worker:0.9.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sp-worker/0.9.0/kxi-sp-worker.0.9.0.qpk
Coordinator registry.dl.kx.com/kxi-sp-coordinator:0.9.0

[NEW] Stream Processor now supports arbitrarily nested namespaces to include operators

[NEW] API added to allow extract of metrics and status for Stream Processor pipelines

[NEW] Pipeline recovery from Controller fault. Workers die after configurable timeout rather than right away and workers retry connection to Controller host:port periodically after disconnect

[NEW] Support added for chunked file reading. Chunked reading will leverage a concept called 'partials'. Readers will indicate if they write 'partials' into the stream. This will be identified by a new optional emit partials discovery function in the reader's plugin interface. By default this function will return false. If a reader emits partials, the next operator in the pipeline can indicate it buffers partials. If a plugin can buffer partials the partials will be sent directly to the operator. If it cannot read partials, a buffer operator will automatically be inserted in the pipeline that will buffer all partials until a full message is available. If an operator emits partials, it must implement a combine function that merges partial results. This combiner will be made available to the downstream operator.

Proposed API additions:

generatesPartials - This operator will output partial results emitsPartials - Should partial results be passed downstream if the next operator can handle it? combinePartials - Given n sets of partials, combine the data processesPartials - Indicates an operator can process partial data usePartials - Indicates to the operator that data will be provided as partials

The result will allow readers to emit partial results. If decoders can process partials directly they will be given the partials function and partials when available. If they cannot, data will be buffered until a full message is emitted. This will also allow reduce operations to be distributed with partial results being emitted to a combine stage and being buffered before they are finally emitted.

[NEW] API for retrieving the DAG spec from a running pipeline to manifest in the UI

[NEW] Pipeline teardown API - REST clients such as the UI should be able to request the teardown of a pipeline to the coordinator. Teardown should request a teardown from all workers. After the workers cleanly exit, the controller and worker resources need to be deallocated. This includes removing pods, stateful sets, services, etc.

[NEW] Coordinators now use heartbeats to identify pipelines that are still active

[NEW] Kubernetes workloads to StatefulSets, for future integration with RT as well as better persistence of checkpointing

[NEW] Kafka reader 'fromBeginning' / 'fromEnd' usability added

[NEW] Length error running passing some atoms to a pipeline

.qsp.run
    .qsp.read.fromCallback[`publish; `publish]
    .qsp.map[`myMap; {x+1}]
    .qsp.write.toConsole[]

If you pass this pipeline the atom 1, it works fine

publish 1 But if you call it on something with more than one character, you get a length error in .spwrite.console.write

publish 100 I'm not sure what the error is here. Can pipelines accept atoms, or is the issue that .sp.console.write isn't handling them? If pipelines can't accept atoms, it should error earlier

[NEW] Stream Processor now includes CSV Decoder - operators and basic type casting

[FIX] .sp.push isn't working for operators with foreign values as parameters

[FIX] Kafka commit offset on checkpoint

[FIX] Kafka library was not reading from the start of a partition

[FIX] .qsp.run was not starting all parent pipelines

[FIX] Added cleanup callback on pipeline teardown

[FIX] .qsp.filter should verify the type of the return value

[FIX] Runtime fails to compile when operations have the same ID

[FIX] isAsync dict is was incorrectly formed if two plugins had the same ID

Data Access

microservice Docker qpk
Data Access Process(DA) registry.dl.kx.com/kxi-da:0.9.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-da/0.9.0/kxi-da.0.9.0.qpk
Data Access Operator registry.dl.kx.com/kxi-da-operator:0.9.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-da-operator/0.9.0/kxi-da-operator.0.9.0.qpk

[NEW] DA now return failure codes when a query is unsuccessful

[NEW] Definition and implementation of what metrics the DA microservice will expose

[NEW] React to data lifecycle events

[NEW] Determine, fulfill, and report temporal purview of process at start-up

Storage Manager

microservice Docker qpk
Storage Manager registry.dl.kx.com/kxi-sm:0.9.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sm/0.9.0/kxi-sm.0.9.0.qpk
End of Interval(EOI) registry.dl.kx.com/kxi-sm-eoi:0.9.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sm-eoi/0.9.0/kxi-sm-eoi.0.9.0.qpk
End of Day(EOD) registry.dl.kx.com/kxi-sm-eod:0.9.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sm-eod/0.9.0/kxi-sm-eod.0.9.0.qpk
Database Migration(DBM) registry.dl.kx.com/kxi-sm-dbm:0.9.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sm-dbm/0.9.0/kxi-sm-dbm.0.9.0.qpk

[NEW] Support for pushing data to S3 via SM

[NEW] Added ObjStor tier option for initial import

[FIX] Fix wrong RDB and IDB purview for freshly imported DB

[FIX] Kurl updates for on-prem DBs

[NEW] Utility to copy data to object storage

[NEW] Updated partition layout options w.r.t. object storage

[NEW] Migration integration with EOD

[NEW] Add tier compression

[NEW] Generic/custom tickerplant as EMS substrate

Service Gateway

microservice Docker qpk
Gateway registry.dl.kx.com/kxi-sg-gw:0.9.1
Resource Coordinator registry.dl.kx.com/kxi-sg-rc:0.9.1 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sg-rc/0.9.1/kxi-sg-rc.0.9.1.qpk
Aggregator registry.dl.kx.com/kxi-sg-agg:0.9.1 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-sg-agg/0.9.1/kxi-sg-agg.0.9.1.qpk

[NEW] Surface assembly info and table schemas via SG's getMeta API

[NEW] Added SG asymmetric DAP mode

[NEW] Added Health and Readiness checks for Service Gateway

[NEW] Added request timeout feature - detection and failure.

[NEW] Metadata retrieval API for SG/DA

[NEW] Support for custom code loading in SG and DA

[FIX] Fix to reconnection logic for coordinator when connection is dropped

[NEW] Added Health and Readyness checks for Service Gateway

[NEW] Added request timeout feature - detection and failure.

[NEW] Metadata retrieval API for SG/DA

[NEW] Support for custom code loading in SG and DA

[FIX] Fix to reconnection logic for coordinator when conection is dropped

[FIX] Aggregator sidecar erroring when trying to register with discovery

[FIX] If discovery is up, but then getServices() fails, we did not retry

User Interface

microservice Docker Helm Chart
Dashboards registry.dl.kx.com/kxi-gui-dash:0.9.0 https://nexus.dl.kx.com/repository/kx-insights-charts/gui-dash-0.9.0.tgz
Gateway registry.dl.kx.com/kxi-gui-gateway:0.9.0 https://nexus.dl.kx.com/repository/kx-insights-charts/gui-gateway-0.9.0.tgz
Data registry.dl.kx.com/kxi-gui-data:0.9.0 https://nexus.dl.kx.com/repository/kx-insights-charts/gui-data-0.9.0.tgz

Machine Learning

microservice package
ML Registry https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-registry/0.8.0/kxi-ml-registry.0.8.0.qpk
ML Analytics https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-ml-analytics/0.8.0/kxi-ml-analytics.0.8.0.qpk

[NEW] ML Analytics - A centralized location for all machine-learning analytics produced by KX

[NEW] ML Registry - The ML Registry defines a centralized location for storing versioned machine-learning models and advanced analytics alongside parameters, metrics and other important artifacts data scientists to store information related to their machine-learning workflows.

Miscellaneous

microservice Docker package
Metrics Sidecar registry.dl.kx.com/kxi-sidecar:0.8.0
OpenAPI SDK Generator registry.dl.kx.com/kxi-openapi-codegen:0.8.0 https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-openapi-codegen/0.8.0/kxi-openapi-codegen-0.8.0-shaded.jar