Skip to content

kdb Insights Enterprise - Release Notes

This page details the release notes for the latest series of kdb Insights Enterprise releases. Key themes include improved access controls, usability and faster ingestion of real-time data into the database. To find the release notes for an older release, refer to the previous releases page.

1.8.3

Release Date

2024-02-06

Improvements

  • Python Cloud Readers

    Python cloud readers now support polling for new matches to a glob pattern with the watch parameter. For more information on the watch parameter, refer to the Readers documentation.

  • Packaging Image

    The Packages image has been upgraded to run python:3.11.7-alpine. For information on using Packages, refer to the Packages documentation.

Fixes

Expand to see the full detail of resolved issues here

Packaging

  • An issue that caused the .kxi.udfs.list.all[] function to return a length error when called has now been fixed. For more information on this function, refer to the User-Defined Functions API documentation.

  • User-defined functions (UDFs) in the Pipeline screen are now displayed. Previously, the functions would be saved in the system, but not visible in a pipeline. This issue has been resolved.

Security

  • Various CVEs were remediated as part of this release.

Third-party Dependencies

kdb Insights Enterprise 1.8.3 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.8.3 release of the standalone infrastructure installation scripts.

Artifacts

Type Location
Enterprise insights-1.8.3.tgz
Operator kxi-operator-1.8.2.tgz
CLI kxicli-1.8.3-py3-none-any.whl
C interface kxi-c-sdk 1.8.0
RT Python interface kxi-rtpy-1.8.0
Java interface kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.7.0
Infrastructure kxi-terraform-1.8.0.tgz

1.8.2

Release Date

2024-01-23

Fixes

Expand to see the full detail of resolved issues here

Database

  • An issue has been resolved in relation to Batch Ingest. Previously, some data in the Historical Database (HDB) was persisted to disk but it was unqueryable. This has been fixed, and all data ingested into the HDB is now queryable.

Third-party Dependencies

kdb Insights Enterprise 1.8.2 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.8.2 release of the standalone infrastructure installation scripts.

Artifacts

Type Location
Enterprise insights-1.8.2.tgz
Operator kxi-operator-1.8.1.tgz
CLI kxicli-1.8.0-py3-none-any.whl
C interface kxi-c-sdk 1.8.0
RT Python interface kxi-rtpy-1.8.0
Java interface kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.7.0
Infrastructure kxi-terraform-1.8.0.tgz

1.8.1

Release Date

2024-01-16

Improvements

  • Encryption of data in transit with OpenShift

    The encryption of data in transit is now supported when deploying kdb Insights Enterprise using OpenShift.

  • Stream Processer CSV Decoder Improvements

    The Python CSV decoder now supports the first header option. This should be used when only the first batch of your dataset contains a header in the CSV file. See Decoders for information on the CSV decoder configuration.

Fixes

Expand to see the full detail of resolved issues here

Packaging

  • Resolved an issue where .kxignore files were incorrectly read as bytes instead of strings when compressed on Ubuntu for Windows.

Stream Processor

  • Stream Processor Beta features configuration has been fixed. When deploying the SP Helm chart, setting betaFeatures: true wasn't correctly enabling Beta features.

  • A fix has been introduced in the UI to ensure changes to the position of an SP node in a pipeline are now correctly saved.

  • When creating a pipeline in the UI, the Stream Processor S3 writer now allows for proper writing to an object storage bucket. Previously, this could error with Some nodes have errors, fix them before trying to deploy this pipeline.

  • Previously, the CSV decoder could drop messages when paired with a file reader (Amazon S3, Google Storage, Azure Storage) where chunking was enabled and the CSV decoder header was set to always. This issue has been resolved. See Decoders for information on the CSV decoder configuration.

  • SP Cloud Readers no longer error when there are no files matching the expected pattern. Previously, a cloud reader using glob pattern matching would prevent the pipeline from starting if no files matched. In particular, this would occur when trying to use the file watcher feature and no files matched as of the pipeline starting.

  • The delimiter select field in the Stream Processor CSV node has been updated to allow any delimiter to be used. Previously, there was a restriction on the number of characters that could be submitted, preventing entry of \t or any other delimiter.

  • Resolved the problem of all window nodes triggering their jobs at the same time, despite their configuration settings. This would occur in setups with pipelines with multiple window nodes, or separate pipelines that read and write to same RT stream. See Windows for more details.

Security

  • Various CVEs were remediated as part of this release.

Third-party Dependencies

kdb Insights Enterprise 1.8.1 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.8.1 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.8.1.tgz
Operator kxi-operator-1.8.1.tgz
CLI kxicli-1.8.0-py3-none-any.whl
C interface kxi-c-sdk 1.8.0
RT Python interface kxi-rtpy-1.8.0
Java interface kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.7.0
Infrastructure kxi-terraform-1.8.0.tgz

1.8.0

Release Date

2023-12-11

New Features

  • Data Encryption

    Data in transit within kdb Insights Enterprise can now be encrypted, providing enhanced security. This feature meets compliance and regulatory requirements and protects against cyber-security attacks. For further details on how to enable encryption via the CLI, see Encryption of data in transit.

  • OpenShift Container Platform (OCP) Support

    kdb Insights Enterprise can now be deployed on premise using the OpenShift Container Platform (OCP). This expands on the deployment topology options of kdb Insights Enterprise, and enables customers to take advantage of its capabilities without having to run in a public cloud. For further details, see OpenShift infrastructure prerequisites.

    Note

    See here for details on an issue with Openshift upgrades.

  • Azure and GCP Parquet Reader Support

    The kdb Insights Parquet Reader functionality has been extended to support Azure and GCP object storage. Users can now read, decode, and watch Parquet files stored on Azure and GCP, as well as AWS. These enhancements are available in the UI, the q API and the Python API.

  • HTTP Publishing

    kdb Insights Enterprise has introduced a REST interface to publish data over HTTP. This expands the interoperability of kdb Insights Enterprise for ingesting data. Note that file size is limited to 10MB per publish. See Upload via HTTP for more details.

  • Python Interface to Reliable Transport

    A new Python interface allows you to publish data to Reliable Transport from your own Python application in Docker, or on Linux or Windows. See here for details.

  • New Database Preview API

    You can now query a sample of your data from the database to understand your data/schema, construct more complex queries, or build visuals on the full data set. See here for details.

Improvements

  • Autoscaling Service Gateways

    The Service Gateway and Aggregator can now autoscale if CPU exceeds your configured threshold. The default is to trigger scaling when CPU exceeds 80% for 5 seconds, with a default stabilization period of 300 seconds. All of this is configurable in your values file. See here for details.

  • CLI Enhancements

    The kdb Insights Enterprise CLI contains a number of security and user experience improvements added:

    • Browser-based authentication for users, and the ability to authenticate even without a browser
    • Ability to authenticate once and still straddle administrator and developer views
    • Token caching & refreshing to support the authentication methods
    • Use of profiles to allow for concurrent sessions in different environments

    See CLI Authentication for details.

  • UI Enhancements

    • When editing a pipeline node, you are now reminded to apply your current changes before moving away to perform other operations. This avoids the potential loss of changes when clicking on to another node.

    • Ability to view the rolled-up status of a pipeline group when there are multiple pipelines/replicas.

    • Undo and Redo buttons have been added to the Pipeline editor.

    • Pipeline settings have been moved into a new ‘Settings’ tab on the Pipeline page. See here for details.

    • For tables marked in a foreign key relationship, further validation has been added upon startup to check that these exist and fail fast if any are missing.

    • The autocomplete widget in code editors now only appears when manually triggered with Ctrl + Space.

  • Stream Processor

    • Cloud file readers in the Stream Processor now support watching for new files. You can now provide a pattern expression to search for file names. When watching is enabled, the file reader will poll for new files that match the expression. For full details on file watching, see the Amazon S3 reader, the Azure Blob storage reader and the Google Cloud storage reader.

    • You can now deploy replicas of pipelines. When deploying, select the “Advanced” option to configure a “Replicas” input field. The environment will deploy the configured number of replicated pipelines identical in configuration with a different ID. See here for details.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Improved startup performance of HDB DAPs when loading late data from IDB tier mount.

  • Storage Manager resiliency improvements have been made to improve how SM cleans up aborted transactions after a failure event.

  • Improved validation for schemas in the UI. Rules now validate against invalid configuration combinations with attributes, sorts and reserved words.

  • Improved failure recovery for object storage in Storage Manager. This resolves a case where EOI would not complete and would report SM Flushing filesystem with [directory] errors.

Reliable Transport

  • Logging for LEADER/FOLLOWER updates have been changed to DEBUG level to reduce log noise.

Stream Processor

  • Previously, it was possible that FINISHED SP pipelines using multiple partitioned workers could spawn extra zombie workers. This has been resolved.

  • Pipeline quick tests no longer fail if they contained a kdb Insights Stream or kdb Insights Database reader or writer.

  • Previously it was not possible to maintain state of any object that contained a kdb guid type. This is now fixed. However, if a rollback to an earlier version is required after an upgrade, a rollback step should be run so that on-disk checkpoints can be converted to a format compatible with earlier versions.

  • Fixed an issue where the csv decoder, with option header=first, and paired with an unbounded streaming reader, was incorrectly dropping the first record in every stream batch.

  • Fixed a determinism gap when reading objects from cloud storage using a glob pattern. In a recovery scenario, it was possible to receive duplicate data.

  • Fixed a bug where unpopulated column of type 0h (i.e. list) was not getting filled in with blank entries correctly.

  • If a pipeline is deployed as part of an assembly, .qsp.write.toDatabase now defaults to writing to that assembly. This means the assembly name can be omitted when calling .qsp.write.toDatabase, for example, .qsp.write.toDatabase[trade; .qsp.use`directWrite!(::;1b)].

  • Fixed a bug where in a single path pipeline, the existence of more than one emitter resulted in that same number of combine nodes being added.

Upgrades

  • Resolved a previously-published known issue where upgrades could occasionally fail with an error when running kxi install upgrade. This occurred due to a race condition in the 1.7.0 version of the CLI when trying to replace the CRDs.

File Watchers

  • When using the file/cloud readers in a non-file watching scenario, files that were not getting chunked did not have their state checkpointed. This meant that during a recovery scenario, duplicate data would be generated for those files.

  • An issue has been resolved that could cause the file watcher to lose data at the boundaries when chunking was set to auto.

Security

  • Various CVEs were remediated as part of this release.

User Interface

  • Resolved the issue where the resizing of a browser window threw the error Error: Cannot read properties of undefined (reading 'getBoundingClientRect') or this.getElements()[0] is undefined.

  • Resolved an issue where certain Python libraries such as spacy, tensorflow, keras did not work in the Scratchpad.

  • Postgres and SqlServer nodes no longer incorrectly have a password field.

  • Resolved the issue where the Views page in the UI occasionally froze when performing certain operations using the map.

  • The instances where the error "Failed to compare saved assembly [assembly name] to deployed version" were erroneously triggered are now resolved.

  • Resolved the issue where resizing your browser window throws "Error: Cannot read properties of undefined (reading 'getBoundingClientRect') or this.getElements()[0] is undefined".

  • Duplicate filters are no longer being reported after creating then saving a filter.

  • Navigating to a Dashboard/Screen from Actions is now working in Views. Previously, you could receive the error "Error: Cannot read properties of undefined (reading 'navigate')" or be redirected to the homepage.

  • Creating a View with a Datagrid, copy it to a second View and pasting, and then returning to your original View would cause your changes to be lost. This has now been resolved.

  • Fixed the incorrect JSON file format that was causing issues when trying to download pipeline logs from the UI.

Third-party Dependencies

kdb Insights Enterprise 1.8.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.8.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.8.0.tgz
Operator kxi-operator-1.8.0.tgz
CLI kxicli-1.8.0-py3-none-any.whl
C interface kxi-c-sdk 1.8.0
RT Python interface kxi-rtpy-1.8.0
Java interface kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.7.0
Infrastructure kxi-terraform-1.8.0.tgz

Upgrade notes

  • Encryption of data in transit is turned on by default for fresh installs. When you upgrade kdb Insights Enterprise the CLI gives you the option to turn encryption on or off. To change this setting after installation, perform an upgrade.

  • When creating a new database through the kdb Insights Enterprise UI, the isSharded schema property is now being defaulted to true. See here for more details on the implications of partitioned tables.

Known Issues

  • Upgrading kdb Insights Enterprise on OpenShift will report that the insights-on-k8s task fails to install Istio. This can be safely ignored. Please follow the instructions to install Istio as a prerequisite.

  • Using a Stream Processor Amazon S3 reader may fail when running on EKS. To work around this, please set the tenant field to kxinsights.

  • If you have an existing kxi-license secret created from a kc.lic or k4.lic, kxi install setup re-uses this secret, however it does not set the appropriate values in the resulting values.yaml. See the available license types here.

  • Occasionally, installs and upgrades may fail with an error when running kxi install. This can occur due to timeouts with the helm upgrade, giving errors such as:

        timed out waiting for the condition
        context deadline exceeded
    

    Re-run the install or upgrade to resolve this issue.

  • Occasionally, pods may not be schedulable after an upgrade or rollback due to a volume node affinity conflict. This can occur with all previous versions of Insights, and the same resolution applies to all. To resolve this, first create a json containing pod, pvc, pv, and node data:

    kubectl get pods -o json | jq '.["pods"] = .items | del(.items)' >pods.json
    kubectl get pvc -o json | jq '.["pvcs"] = .items | del(.items)' >pvcs.json
    kubectl get pv -o json | jq '.["pvs"] = .items | del(.items)' >pvs.json
    kubectl get nodes -o json | jq '.["nodes"] = .items | del(.items)' >nodes.json
    jq -s '.[0] * .[1] * .[2] * .[3]' pods.json pvcs.json pvs.json nodes.json > pods_pvs.json
    
    jq '{
    pods:[.pods[]| {pod: .metadata.name, pvc: .spec.volumes[]|select(.persistentVolumeClaim != null)|.persistentVolumeClaim.claimName,nodeName: .spec.nodeName}],
    pvcs:[.pvcs[]| {pvc: .metadata.name , pv: .spec.volumeName}],
        pvs:[.pvs[]| {pv: .metadata.name , nodeAffinity:  .spec.nodeAffinity.required.nodeSelectorTerms[0]|.matchExpressions[0]|select(.values[0] != null)|.values[0]}],
    nodes:[.nodes[]| {node: .metadata.name, zone: .metadata.labels."topology.kubernetes.io/zone"}]
    }' pods_pvs.json > pods_pvs_simplified.json
    
    jq '(INDEX(.pvcs[]; .pvc)) as $dict_pvc
    | (INDEX(.pvs[]; .pv)) as $dict_pv
    | (INDEX(.nodes[]; .node)) as $dict_node
    | [.pods[]
    |{pod:.pod,
    pvc:.pvc,
    pv: ($dict_pvc[.pvc]|.pv),
        zone: ($dict_node[.nodeName]|.zone),
        nodeAffinity: ($dict_pv[($dict_pvc[.pvc]|.pv)]|.nodeAffinity)
    }]|group_by(.pod)|
        map({pod: (.[0].pod),
            nodeAffinity: ([.[] | .nodeAffinity]|unique),
            zone:([.[] | .zone]|unique)
        })' pods_pvs_simplified.json > PODS.json
    

    Identify affected pods from events e.g.

    $ kubectl get events
    LAST SEEN   TYPE      REASON                   OBJECT                           MESSAGE
    21m         Warning   FailedAttachVolume       pod/dfx-assembly-dap-da-0    pod didn't trigger scale-up: 1 node(s) had volume node affinity conflict
    

    Find the nodeAffinity of the affected pod.

    $ jq '.[]|select(.pod == "dfx-assembly-dap-da-0")|select(.nodeAffinity != [null])' PODS.json
    {
    "pod": "dfx-assembly-dap-da-0",
    "nodeAffinity": [
            null,
            "eu-west-1c"
        ],
    "zone": [
            null
        ]
    }
    

    Find pods running in the target zone of the affected pod that have no nodeAffinity.

    $ jq '.[]|select(.nodeAffinity == [null])|select(.zone == ["eu-west-1c"])|.pod' PODS.json
    "insights-kxi-package-manager-5776fbc46-rzfqz"
    "insights-qe-resource-coordinator-0"
    "insights-resource-coordinator-0"
    

    Delete these pods.

    kubectl delete pods insights-resource-coordinator-0 insights-kxi-package-manager-5776fbc46-rzfqz insights-qe-resource-coordinator-0
    

    Kubernetes should now schedule the affected pod and the manually deleted pods.

  • With entitlements enabled, the Query tab in the UI will not return data as there are issues with resolving the correct database name when checking entitlements. In order to work around this issue, you should create a second entitlement for each database / assembly.

    For example, if you create and entitle an assembly called dfx-assembly, you should also create an identical entitlement called dfx-assembly-qe. This -qe entitlement should be kept in-sync when updating the main one.

    This restriction will be removed in a future release.

  • qsql requests, made either through the Q tab (Query environment) in the UI or via REST to /qsql, do not work with entitlements. The .kxi.qsql API should be disabled if using entitlements. This is disabled by default in kdb Insights Enterprise. See here for more details.