Skip to content

kdb Insights Enterprise - Previous Release Notes

This page outlines release notes from past releases. For the most recent release, refer to the latest release page.

1.4.5

Release for kdb Insights Enterprise.

Release Date

2023-04-26

Improvements

[IMPROVEMENT]: If publishing a message larger than 1GB, the publisher will now receive a messsage too large error. Users can catch the error (need more clear detail) to avoid getting an error at runtime. This would previously cause downstream subcriber processes to run out of memory during processing.

[IMPROVEMENT]: SQL queries that filter by time but do not explicitly select the time column are now supported. This would previously error.

Fixes

[FIX]: Fixed an issue where rapid successive queries using qsql such as those from Views in the kdb Insights UI could return query results out of order.

[FIX]: Schema conversions are now more robust on network based file systems (ex. Rook Ceph). Previously, this could result in a EXDEV. OS Reports: Invalid cross-device link error in the Storage Manager.

[FIX]: Window nodes in the Stream Processor can now accept dictionaries as input batches. Previously, incoming dictionaries would be merged with input batch in the buffer, instead of appending to a buffered list of dictionaries.

Third-party dependencies

kdb Insights Enterprise 1.4 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.4.5.tgz
Operator kxi-operator-1.4.2.tgz
CLI kxicli-1.4.2-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.1.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly created using a version prior to 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

Licensing

A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below.

klic license renew $LICENSE_UUID --version=latest
env QLIC=$(pwd) klic license fetch $LICENSE_UUID

Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.

printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml
kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Monitoring

We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes. 

If you wish to continue with the higher frequency you can take one of the following steps:

-  Update the interval field within their charts.

    ```yaml
    metrics:
      serviceMonitor:
        interval: "30s"
    ```

- Override the chart values using the global value within insights/values.yaml.

    ```yaml
    global:
      metrics:
        serviceMonitor:
          interval: "30s"
    ```

- Update your assembly yaml, to override the default.

    ```yaml
    spec:
      metrics:
        enabled: true
        serviceMonitor:
          enabled: true
          interval: 10s
    ```

- Update your default within the Assembly Resource - This is generated using the insights/values.yaml.

    ```yaml
    kxi-operator:
      config:
        sidecar:
          metrics:
            serviceMonitor:
              interval: 30s
    ```

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • Publishing of a single message larger than 1GB is not supported.
  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored. 

"[] Failed to compare saved assembly  to deployed version, error: type"

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

  • When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

  • Screen readers will not read out node information represented in a pop-up when hovered over, and may freeze when attempting to read a large amount of content.

  • Unable to navigate up and down cells and read all table content when navigating using only a keyboard.

  • The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml. 
keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

Backward Compatibility

If you are upgrading from a version earlier than 1.4.0 you must also read each of the relevant release notes to ensure you know all the changes that will affect the upgrade.

The most pertinent upgrade notes from previous versions are listed below:

1.4.4

Release for kdb Insights Enterprise.

Release Date

2023-04-20

Improvements

[IMPROVEMENT]: The kdb Insights Database now supports a REST interface for loading historical datasets into an existing database. You can now perform a historical batch ingest using a simple REST API. For more details, see batch ingestion.

[IMPROVEMENT]: Query requests that supply a log correlator (logCorr) will now see that correlator in the Service Gateway logs, allowing for easier traceability in query execution and an improved debugging experience. Previously a random correlator was used in the Service Gateway logs.

[IMPROVEMENT]: Added significant improvements for accessibility using keyboard navigation and/or screen readers in the UI.

  • Users can exit a text area in a code block on the Query page by first pressing "CTRL+M" before pressing the "Tab" key.

  • The entity tree menu button can now be opened when hovering your mouse over the entity tree. Arrows can then be used to navigate.

  • "Code View" on the Schemas page can now be closed with the "Esc" key.

  • The Views and Pipeline pages now allow draggable elements to be inserted/dragged onto the page with the use of your keyboard.

Fixes

[FIX]: Security vulnerabilities patched in the Stream Processor for the following CVEs: CVE-2022-41910, CVE-2022-41902, CVE-2022-41900, CVE-2022-41880.

[FIX]: Additional historical tiers that use object storage in the UI now allow dashes and spaces in file names. Previously they would error for dashes and spaces.

[FIX]: Migrating tiers to object storage could fail with the eventual error SMDBM Migration discontinued due to failure if multiple threads were used.

[FIX]: If an SM client cannot connect to SM after a number of retries, it will now restart the pod to try to automatically recover. This resolves connection issues in DAPs when there are network connectivity issues.

[FIX]: Reading of files from Azure Storage larger than ~1MB now successfully writes to a kdb Insights database. Previously this was erroring due to an issue in the chunking of larger files.

[FIX]: The Service Gateway was creating duplicate connections to Resource Coordinators while they were waiting to come online. This was resulting in a large number of redundant connections being opened between the services. In large deployments, this could result in connection errors in the Resource Coordinator.

[FIX]: Expression readers now return the result of Python lambdas.

[FIX]: After upgrading Insights from 1.3.x to 1.4.x, clicking a node in a previously-saved pipeline in the UI no longer causes the function definition to flicker and disappear.

[FIX]: Fixed issue with 1.4.0 kxi-terraform scripts, where Terragrunt config was incompatible and was causing deployment failures on AWS.

Third-party dependencies

kdb Insights Enterprise 1.4 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.4.4.tgz
Operator kxi-operator-1.4.2.tgz
CLI kxicli-1.4.2-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.1.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly created using a version prior to 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

Licensing

A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below.

klic license renew $LICENSE_UUID --version=latest
env QLIC=$(pwd) klic license fetch $LICENSE_UUID

Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.

printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml
kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Monitoring

We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes. 

If you wish to continue with the higher frequency you can take one of the following steps:

-  Update the interval field within their charts.

    ```yaml
    metrics:
      serviceMonitor:
        interval: "30s"
    ```

- Override the chart values using the global value within insights/values.yaml.

    ```yaml
    global:
      metrics:
        serviceMonitor:
          interval: "30s"
    ```

- Update your assembly yaml, to override the default.

    ```yaml
    spec:
      metrics:
        enabled: true
        serviceMonitor:
          enabled: true
          interval: 10s
    ```

- Update your default within the Assembly Resource - This is generated using the insights/values.yaml.

    ```yaml
    kxi-operator:
      config:
        sidecar:
          metrics:
            serviceMonitor:
              interval: 30s
    ```

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.
  • The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored. 
"[] Failed to compare saved assembly  to deployed version, error: type"

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

  • When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

  • Screen readers will not read out node information represented in a pop-up when hovered over, and may freeze when attempting to read a large amount of content.

  • Unable to navigate up and down cells and read all table content when navigating using only a keyboard.

  • The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml. 
keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

Backward Compatibility

If you are upgrading from a version earlier than 1.4.0 you must also read each of the relevant release notes to ensure you know all the changes that will affect the upgrade.

The most pertinent upgrade notes from previous versions are listed below:

1.4.3

Release Date

2023-03-31

Improvements

[IMPROVEMENT] We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes. 

If you wish to continue with the higher frequency you can take one of the following steps:

-  Update the interval field within their charts.

    ```yaml
    metrics:
    serviceMonitor:
        interval: "30s"
    Assemblies
    ```

  • Update your assembly yaml, to override the default.

    ```yaml
    spec:
        metrics:
            enabled: true
            serviceMonitor:
            enabled: true
            interval: 10s
```

  • Update your default within the Assembly Resource - This is generated using the insights/values.yaml.

    ```yaml
kxi-operator:
config:
    sidecar:
    metrics:
        serviceMonitor:
        interval: 30s
```

Fixes

[FIX]: Service Gateway: fixed remote code injection vulnerability in Snake YAML CVE-2022-1471

[FIX]: Queries to the kdb Insights Database now have a default timeout of 60s (previously 30s). This allows for longer running queries to be issued from query clients, such as the scratchpad.

[FIX]: Additional historical tiers in the UI now allow for a retention period to be set on the HDB tier. Previously this was erroring due to a validation issue.

[FIX]: In the UI, the real-time database rollover tier now correctly checks that rollover is less than the interval tier.

[FIX]: Requests to object storage (Azure Storage, AWS S3, Google Cloud Storage) will now backoff and retry if they get a throttle request from the vendor. Jitter will be introduced to the retry to ensure that replicated processes do not all retry at the same time.

[FIX]: Setting a column type to Any now correctly uses an anymap type. This was previously reset to an unselected value.

[FIX]: The REST Server's heartbeat logs are now set to TRACE to reduce log output (previously they were logged as INFO logs).

[FIX]: Pasting into a pipeline node’s configuration no longer duplicates the node.

[FIX]: Resolved issue where some pipelines with merge nodes would error with 'missing edge'.

[FIX]: Comments can now be added to the last line of the Scratchpad without causing it to error.

[FIX]: The Scratchpad now has more memory allocated for ad-hoc queries in the kdb Insights Enterprise query panel.

[FIX]: Stream Processor file reader nodes (Azure, AWS S3, File and Google storage) would only process the first file presented even when multiple files were present.

[FIX]: Object storage tiers can now be added to a database via the kdb Insights Enterprise UI.

Third-party dependencies

kdb Insights Enterprise 1.4.3 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.3 release of the standalone infrastructure installation scripts.

Artifacts

type location

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored. 

"[] Failed to compare saved assembly  to deployed version, error: type"

  • Please ensure all pipelines have different names. Two pipelines with the same name may not be deploy correctly.

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml. 

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

Backward Compatibility

If you are upgrading from a version earlier than 1.4.0 you must also read each of the relevant release notes to ensure you know all the changes that will affect the upgrade.

The most pertinent upgrade notes from previous versions are listed below:

1.4.2

Release for kdb Insights Enterprise.

Release Date

2023-03-17

Fixes

[FIX] Resource utilization in the UI Scratchpad has been greatly improved. This fix along with a larger default memory allocation provide greater Scratchpad process stability.

Third-party dependencies

kdb Insights Enterprise 1.4.2 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.4.2.tgz
Operator kxi-operator-1.4.0.tgz
CLI kxicli-1.4.1-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored. 

"[] Failed to compare saved assembly  to deployed version, error: type"

  • When creating a pipeline the user interface does not stop you from creating it with the same name as an existing pipeline, however two pipelines with the same name may not be deploy correctly. Please ensure all pipelines have different names.

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml. 

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

Backward Compatibility

If you are upgrading from a version earlier than 1.3.3 you must also read each of the relevant release notes to ensure you know all the changes that will affect the upgrade.

The most pertinent upgrade notes from previous versions are listed below:

1.4.1

Release for kdb Insights Enterprise.

Release Date

2023-03-14

Fixes

[FIX] In 1.3.3 and 1.4.0, if the Storage Manager falls behind on EODs for more than a full day, IDB links could be written incorrectly when late data was present. If this issue is encountered, manual data changes are required to recover the database including backup of the stream logs from the date of the failure. Please contact KX Support for help with recovery.

This issue is characterized by a log statement from the Storage Manager as follows: "Unexpected condition: EOXa watermark date YYYY.MM.DD is less than HDB stream date YYYY.MM.DD, setting it to match HDB stream date". The IDB may also start reporting "No such file or directory" on ordinal partitions.

Third-party dependencies

kdb Insights Enterprise 1.4.1 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.4.1.tgz
Operator kxi-operator-1.4.0.tgz
CLI kxicli-1.4.1-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database - this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • If end-of-day processing is interrupted and remains incomplete until the next EOD period, this prevents further EODs from completing and requires manual recovery.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored. 

"[] Failed to compare saved assembly  to deployed version, error: type"

  • Please ensure all pipelines have different names. Two pipelines with the same name may not be deploy correctly.

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml. 

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

Backward Compatibility

If you are upgrading from a version earlier than 1.3.3 you must also read each of the relevant release notes to ensure you know all the changes that will affect the upgrade.

The most pertinent upgrade notes from previous versions are listed below:

1.4.0

Release for kdb Insights Enterprise.

Release Date

2023-03-03

New Features

Introducing Pythonic pipeline development in the UI, dynamic logging notifications providing direct access to root cause of underlying issues, an enhanced user experience for database configuration and additional security enhancements.

[NEW] Pipelines

  • You can now use Python in the pipeline builder in the UI to perform aggregations and analytics.

  • Ingest large batches of data directly into the historical database has been moved out of beta.

    • Use fewer resources. Memory and disk resources greatly reduced during ingest.
    • Select the Write Direct to HDB field in the Database Writer in the UI, or use the q and Python APIs.

[NEW] User Interface

  • Improvements to the observability user experience with two new icons in top right hand-side of the UI banner:

    • A notification icon shows a count of new log messages. This ensures you are notified of any new error messages wherever you are in the UI. You can also click on the icon to show more details of the message, these are displayed in the right-hand third of the screen.

    • A settings icon enables you to update the application logging level for future log records written by the kdb Insights Enterprise components.

  • You can now share the URL of a View by selecting the view on the left-hand toolbar, right-clicking and choosing the new Share button. This URL just shares the content of the View and not the toolbars (e.g. it does not show the left-hand toolbar or the Design/Preview panel at the top).

[NEW] Databases

A new version of our Database Configuration page is now available, under a beta flag, that provides an enhanced user experience. It can be toggled on and off from the Settings option in the main toolbar. We recommend that you use the existing Database Configuration age for all running databases, but please try the new user interface out and give us your feedback.

[NEW] Packages

The user interface now allows you to configure which Packages and their associated Custom APIs are used within individual databases, this had previously only been possible through kdb CLI deployments.

[NEW] Security Improvements

  • Keycloak has been updated to version 19. This impacts the following:
    • The Keycloak Admin console has been changed
    • Logout screen provides a redirection to the login page

Improvements

Keycloak realm

In this release, the handling of the Keycloak realm has been improved. This realm contains the role definitions, users and clients. Previously the realm was not being imported by default on install. The role definitions will always be imported as part of an install or upgrade now. The users and clients will only be imported by default on install (kxi install), and not on upgrade.

To make changes to the initClient and initUser on upgrade, you will need to tell the application to import them using the setting below.

keycloak:
  importUsers: true

Reliable Transport

The kdb Insights Enterprise transport layer has adopted a version of Raft called NuRaft. This improves the fault tolerance and performance of kdb Insights Enterprise. NuRaft is an open source version of Raft. Migrating to NuRaft also provides kdb Insights Enterprise with additional capabilities for future enhancements to Reliable Transport.

Third-party dependencies

kdb Insights Enterprise 1.4.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts .

Artifacts

type location
Enterprise insights-1.4.0.tgz
Operator kxi-operator-1.4.0.tgz
CLI kxicli-1.4.0-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database - this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps .

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Supported Python versions in the CLI

Support for Python 3.7 has been dropped in the CLI.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • If end-of-day processing is interrupted and remains incomplete until the next EOD period, this prevents further EODs from completing and requires manual recovery.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored. 

"[] Failed to compare saved assembly  to deployed version, error: type"

  • When creating a pipeline the user interface does not stop you from creating it with the same name as an existing pipeline, however two pipelines with the same name may not be deploy correctly. Please ensure all pipelines have different names.

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml. 

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

Backward compatibility

If you are upgrading from a version earlier than 1.3.3 you must also read each of the relevant release notes to ensure you know all the changes that will affect the upgrade.

The most pertinent upgrade notes from previous versions are listed below:

1.3.4

Release for kdb Insights Enterprise.

Release Date

2023-03-13

Fixes

[FIX] As of 1.3.3, if the Storage Manager falls behind on EODs for more than a full day, IDB links could be written incorrectly when late data was present. If this issue is encountered, manual data changes are required to recover the database including backup of the stream logs from the date of the failure. Please contact KX Support for help with recovery.

This issue is characterized by a log statement from the Storage Manager as follows: "Unexpected condition: EOXa watermark date YYYY.MM.DD is less than HDB stream date YYYY.MM.DD, setting it to match HDB stream date". The IDB may also start reporting "No such file or directory" on ordinal partitions.

Artifacts

type location
Infrastructure kxi-terraform-1.3.1.tgz
kdb Insights Enterprise insights-1.3.4.tgz
Operator kxi-operator-1.3.1.tgz
CLI kxicli-1.3.1-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0

Batch Ingest Known Issues

  • In the unlikely event of two batch ingests being done within an end of interval (EOI) event (for example, a 10 min window) a one hour delay would occur for the subsequent ingest, but there would not be any data loss, just a delay.
  • If the Storage Manager (SM) crashes during a batch ingest, the ingest will fail. On SM recovery, it will not try again and must be restarted, but no clean-up is required.

Upgrade Notes

  • When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Resource Coordinator

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" can be misleading in the timeout message. It should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

Python Scratchpad for Queries

  • If the script consists of multiple statements, the console does not print the output of the last evaluation, and instead it only displays the expression before it is evaluated.

S3 Import Wizard

  • If you select the source, S3, and click Next, no error will be thrown, which is invalid. At least one file needs to be present.

Keycloak Config CLI

We recommend that you enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default Password Policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the Insights values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

If you currently set initUser.enabled=true or initClient.enabled=true within your own values.yaml, you may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where a Keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, you must set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly blockSize Changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

  • if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
  • this has the highest performance, but has no RAM limit on received data
  • if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
  • the smaller this number is set to, the worse ingest performance but stronger RAM limits; this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1
  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation does not rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, you will not be redirected back to the login screen.
  • If you were logged out due to inactivity, you might see a "Logout failed" error.

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}.

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 and kdb Insights Enterprise 1.2.0 if you're upgrading from versions earlier than 1.1.0 or 1.2.0, for notes about:

1.3.3

Release for kdb Insights Enterprise.

Release date

2023-01-17

New features

Functionality for improved ingestion, efficiency and reduction of resources. Take advantage of an IDE to develop your databases, pipelines and query interfaces.

[NEW] Improved ingestion

  • [Beta] Ingest large batches of data directly into the historical database:

    • Use fewer resources. Memory and disk resources greatly reduced during ingest.
    • Select the Direct Writing field in the Database Writer in the UI, or use the q and Python APIs.

    See details here for known issues.

[NEW] Writing data to Amazon S3

  • You can now write data to Amazon S3 by either:

[NEW] Improved IDE capabilities

  • Build YAML files and upload them using kdb Insights CLI:

    • Speed up development workflow, improve productivity.
    • Access to the Kubernetes control plane is no longer needed. This is now handled for you by the CLI.
    • Assembly management using CLI.

  • If you have access to KX Analyst, you can use it to develop functions. You can now:

    • Enjoy a rich IDE development environment with syntax highlighting for YAML files.
    • Query kdb Insights Enterprise data using REST API or query interface.
    • Upload using the CLI or as part of CI/CD pipeline.
    • Build a complete SDLC that harnesses the flexibility of KX Analyst against kdb Insights Enterprise data.

    Note

    Contact your KX Sales representative if you are interested in using KX Analyst. In future releases, additional IDEs will become available, including VS Code and Jupyter Notebooks.

Improvements

[NEW] Reliable Transport

  • Reliable Transport's logging default level for the sequencer and merger has been changed from DEBUG to INFO.

Fixes

[FIX] Following a hard reset of the Reliable Transport (RT), the log files in the out folder are truncated rather than deleted. This means that the replicated log files for all subscribers to the RT instance will be garbage collected once the RT archiver starts after the reset. This prevents a hard reset from causing subscriber PVCs to fill up due to old/corrupt log files remaining.

Artifacts

type location
Infrastructure kxi-terraform-1.3.1.tgz
kdb Insights Enterprise insights-1.3.3.tgz
Operator kxi-operator-1.3.1.tgz
CLI kxicli-1.3.1-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0
RT Bridge rt-bridge-1.4.0-install.sh

Known Issues

Batch ingest

  • In the unlikely event of two batch ingests being done within an end of interval (EOI) event (for example, a 10 min window) a one hour delay would occur for the subsequent ingest, but there would not be any data loss, just a delay.
  • If the Storage Manager (SM) crashes during a batch ingest, the ingest will fail. On SM recovery, it will not try again and must be restarted, but no clean-up is required.

Discovery Service

  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.

Upgrade Notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Terraform Script

Note the version. GKE cluster cannot be deployed using a 1.3.0 version of the script; a 1.3.* version of the client scripts is needed to use Kubernetes v1.22 on GKE.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the Keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the Insights values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

If you currently set initUser.enabled=true or initClient.enabled=true within your values.yaml, you may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where Keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly blockSize changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

  • If unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval. This has the highest performance, but has no RAM limit on received data
  • If set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM. The smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed.

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

If the script consists of multiple statements, the console does not print the output of the last evaluation, and instead it only displays the expression before it is evaluated.

If S3 is selected as the source and Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 and kdb Insights Enterprise 1.2.0 if you're upgrading from versions earlier than 1.1.0 or 1.2.0, for notes about:

1.3.2

Release for kdb Insights Enterprise.

Release Date

2022-12-05

New Features

[NEW] Pipelines

  • [Beta] Compressed files (GZIP) can now be decoded using the q and Python APIs.

  • Multiple file paths can be added to a cloud storage Reader node using the + button in the UI, or via the q and Python APIs.

  • Wildcard support using * is now available when using a single file path with the Amazon S3 Reader.

  • CSV encoding is now available, using a new node in the UI, or via the q and Python APIs.

  • It is now possible to write to Amazon S3 buckets, using a new node in the UI, or via the q and Python APIs.

  • Pipeline lifecycle events can be subscribed to via the q API. This can be used, for example, to invoke a callback once files have been processed by a pipeline.

Improvements

  • Queries will no longer time out after 60 seconds at the gateway. Longer timeouts can be configured in getData API calls using the timeout header option.

Fixes

[FIX] Query

  • An issue where DAPs could enter low-memory mode earlier than they should has been resolved.

Artifacts

type location
Infrastructure kxi-terraform-1.3.0.tgz
Platform insights-1.3.2.tgz
Operator kxi-operator-1.3.1.tgz
CLI kxicli-1.3.0-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within kdb Insights Enterprise values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

Users who currently set initUser.enabled=true or initClient.enabled=true within their own values.yaml may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
    .Values.keycloak.initUser.name
    .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
    .Values.keycloak.initClient.clientId
    .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"
Where keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, the user is required to set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly blockSize changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

  • if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
    • this has the highest performance, but has no RAM limit on received data
  • if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
    • the smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1
  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 and kdb Insights Enterprise 1.2.0 if you're upgrading from versions earlier than 1.1.0 or 1.2.0, for notes about

1.3.1

Release for kdb Insights Enterprise.

Release Date

2022-11-23

Security

[FIX] kdb Insights Enterprise 1.3.1 contains security updates for a number of critical CVEs

Streams and Reliable Transport

[FIX] An issue which could cause RT archiving to fail has been fixed.

Artifacts

type location
Infrastructure kxi-terraform-1.3.0.tgz
Platform insights-1.3.1.tgz
Operator kxi-operator-1.3.1.tgz
CLI kxicli-1.3.0-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights Enterprise values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

Users who currently set initUser.enabled=true or initClient.enabled=true within their own values.yaml may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
    .Values.keycloak.initUser.name
    .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
    .Values.keycloak.initClient.clientId
    .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"
Where keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, the user is required to set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly blockSize changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

  • if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
    • this has the highest performance, but has no RAM limit on received data
  • if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
    • the smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1
  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 and kdb Insights Enterprise 1.2.0 if you're upgrading from versions earlier than 1.1.0 or 1.2.0, for notes about

1.3.0

Release for kdb Insights Enterprise.

Release Date

2022-10-26

Features

[NEW] User Interface

  • Items in the page tree have been renamed for clarity and consistency with the CLI.
  • Data is now known as Assemblies
  • Explore is now known as Queries
  • Reports is now known as Views

[NEW] Pipelines

  • An "Auto" setting for Parse Strings in the Apply Schema node has been added. The column will be parsed if the input is textual (strings or bytes), but the output type is non-textual.

  • Data types are now visible for each column for Test Deploy previews. These can be used to validate the data types and the data shape of Transforms and Maps results.

  • Data types and column names are validated for Database writers as part of a test deploy.

  • Additional Reader validation has been introduced to pipelines, to ensure correctness of settings before deploying.

  • Test deploy previews can now show non-tabular data.

  • Previews are now available for Reader nodes, before data has been decoded.

  • The visibility of a Pipeline's state has been improved in the Overview tab. Warning and error counts can be clicked on to launch a diagnostics tab for that Pipeline.

[NEW] Assemblies

  • It is now possible to see assemblies created using kxi assembly create in the UI. Note that assemblies created and managed through the command-line are not editable via the UI.

  • Additional logging has been added to the Storage Manager, to ease troubleshooting.

  • Schema configuration validation has been improved.

[NEW] Install and Packages

  • The kdb Insights Operator is no longer uninstalled by default by kxi install delete, the command line parameter --uninstall-operator must now be used.

  • Many values for kxi install setup can now be passed via command line parameter, please see the CLI Reference for more detail.

  • Improved support for rolling back to previous versions. Enter a previous version number using the upgrade command kxi install upgrade to roll back.

  • The current kubectl context is now used as the default in all commands, instead of the namespace specified in the cli-config file.

  • kdb Insights Python modules for packaging and machine learning are no longer dependent on libc.musl, and can now be used for development on Windows and Mac.

  • Custom user query APIs and Aggregation functions can be installed and initialised within Data Access Processes and Aggregators as outlined here.

  • Custom user query APIs and Aggregation functions can be installed and initialised within Data Access Processes and Aggregators as outlined here.

[NEW] Observability

  • Reliable Transport diagnostics are now available in diagnostics exports from the UI.

[NEW] Infrastructure

  • The base image for most components has been moved to Rocky Linux.

Fixes

A large number of fixes have gone into kdb Insights Enterprise 1.3.0. Known issues which have been resolved, along with some other highlights, are listed below.

[FIX] Streams and Reliable Transport

  • A hard reset can now be performed on Reliable Transport, to recover from stream corruption. Please contact KX Support for usage instructions.

[FIX] Pipelines

  • Pipelines which have their data cleared when they're torn-down can now be re-deployed with the same name. Previously, this would cause RT sequence number problems, stopping data from flowing.

  • Persistent pop-up errors in the UI after resizing a Map node have been resolved.

  • .qsp.read.fromDatabase can now be used in pipelines.

  • Renaming a pipeline node no longer disconnects its edges.

  • Empty pipelines are no longer deployable.

[FIX] User Interface

  • Validation fixes so that date and time fields can be entered without use of the picker.

[FIX] Java SDK

  • Occasional authentication failures when querying via Java SDK have been resolved. These would previously be seen more frequently on slower connections.

Artifacts

type location
Infrastructure kxi-terraform-1.3.0.tgz
Platform insights-1.3.0.tgz
Operator kxi-operator-1.3.0.tgz
CLI kxicli-1.3.0-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of KX Insights.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the Insights values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

Users who currently set initUser.enabled=true or initClient.enabled=true within their own values.yaml may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
    .Values.keycloak.initUser.name
    .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
    .Values.keycloak.initClient.clientId
    .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"
Where keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, the user is required to set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below 
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

  • The user-defined variables & functions in the Queries tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • The Queries tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Queries tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1
  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 and kdb Insights Enterprise 1.2.0 if you're upgrading from versions earlier than 1.1.0 or 1.2.0, for notes about

1.2.1

Release for kdb Insights Enterprise.

Release Date

2022-09-22

Fixes

kdb Insights Enterprise 1.2.1 is a security update for vulnerabilities in the included images for Keycloak/PostgreSQL.

Artifacts

type location
Infrastructure kxi-terraform-1.2.0.tgz
Platform insights-1.2.1.tgz
Operator kxi-operator-1.2.0.tgz
CLI kxicli-1.2.0-py3-none-any.whl
ODBC Driver kodbc 1.2.0
Java SDK java-sdk 1.2.0
keycloak-server keycloak-server-0.2.1.tgz

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Version 1.2.1 requires a custom storageClassName definition

The addition of support for packages introduces a new storageClassName defined as sharedfiles, this is defined by default in any deployments of kdb Insights Enterprise that rely on the KX provided Terraform scripts for deployment but must be added manually by a user as outlined here.

Alternatively, within the deployment configuration as outlined here, modify the storageClassName to an alternative supported RWX storage option for example rookcephfs.

Keycloak 18

The version of Keycloak used within kdb Insights Enterprise 1.2.0 and 1.2.1 has been upgraded from Keycloak 16 to Keycloak 18.

Please follow the Keycloak 18 migration steps to migrate your Keycloak data prior to upgrading kdb Insights Enterprise from versions older than 1.2.0.

Azure Marketplace upgrade

Upgrading from v1.2.0 Marketplace release has a known issue. Keycloak was upgraded in v1.2.1 which contained non backward compatible changes. The PostgreSQL secret format has changed, and existingSecret from values.yaml has an extra nested key. Please make sure values.yaml has the extra auth nested key, and that the secret containing PostgreSQL passwords, usually called kxi-postgresql, has dropped the postgresql prefix from the data keys.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below 
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data
  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1

  • Torn-down pipelines that include Database or Stream Writers cannot be directly restarted after having their resources removed. Upon redeploy, data will not be published downstream from the pipeline. A workaround is to rename the pipeline. Resources are removed by checking "Clear Pipeline State" during teardown in the UI, or by directly deleting the Kubernetes PVCs backing the pipeline. If either of these things are done, the pipeline must be renamed before relaunching it.

  • .qsp.read.fromDatabase cannot be run in a pipeline to query a kdb Insights Enterprise database. The following example code can be used as a workaround to query data (ex. reference data to be used within a pipeline). 

onResponse:{[hdr;data]
if[0 <> hdr`ac;
    .sp.log.error ("Query failed host=%s, api=%s, code=%n, error=%s";hdr`gw; hdr`api; hdr`ac; hdr`ai)];
publish data
}
.qsp.onStart {
    h:hopen hsym `$.spenv.getReleaseName[],"-sg-gateway:5050";
    res: neg[h] (`.kxi.getData; (enlist `table)!(enlist `instruments); `onResponse; ()!());
}
.qsp.run .qsp.read.fromCallback[`publish] .qsp.write.toConsole[]
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can navigate re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 if you're upgrading from an earlier version, for notes about

1.2.0

Release for kdb Insights Enterprise.

Release Date

2022-09-02

Features

[NEW] Install and Packages

  • Upgrade support{:target="blank"} in the kxi command line tool for easier upgrades. _Please take note of other Upgrade Notes below.

  • You must now provide a namespace when deleting installs using kxi install delete in the CLI.

  • Ability to list, add and delete custom versioned user code via packages to kdb Insights Enterprise via the CLI outlined here.

  • Support within packages for the definition of User Defined Functions (UDFs) written in Python or q for deployment to stream-processor or use in the Explore Window.

[NEW] Pipelines

  • MQTT QoS 0 Reader has been added.

  • Users can run Test deploys in the UI, which provide user feedback on issues when building or editing a pipeline. This includes visualization of the data at each node in the pipeline, and clear indications of which nodes failed.

  • CPU and Memory resources can now be configured from the User Interface to allow rightsizing of data pipelines.

  • Use of User Defined Functions as part of a pipeline.

  • Apply Schema is now available for use after the CSV decoder, so the optional schema parameters for the CSV node have been removed from the Pipelines UI. Users should take care to ensure they're parsing all fields in the Apply Schema node when it's being applied to CSV decoded data.

[NEW] Explore and Query

  • Significant speed improvements to Explore. Query environments are pre-allocated as part of Data/assemblies, and so switching between tabs, assemblies, or instances is now instant. Query environments must be enabled in the Database/Data Access screen, or in assembly configuration, for the assembly to be available in Explore.

  • Filtering and aggregation are now available on the Query tab.

  • There is no need to choose a database when using the Query or SQL options in the Explore window. Label filtering can be used to select particular databases. You can now read from tables with the same name and schema across multiple databases when using the Query or SQL tabs.

  • Import and use of User Defined Functions in pipelines and the Explore Window to allow running of custom code.

  • Console output can now be cleared by right-clicking in the Console and choosing "Clear".

  • The query section at the top of Explore tabs can now be collapsed for a larger code area.

[NEW] Databases

  • CPU and Memory resources can now be configured from the User Interface to allow rightsizing of storage and query services.

  • Redundant replicas possible for all query-path processes, enabling greater query uptime despite process, node, or availability-zone failure.

  • Ability for Data Access tiers (RDB, IDB, HDB) to share compute resources by using mountList within Assembly configuration rather than mountName.

  • Granular assembly deployment status showing the state of deploying components, and assembly deployment errors, are now surfaced in the UI. Mouse-over the assembly's status icon while it's deploying to see the breakdown.

  • Environmental variables can now be configured in the UI for Data Access and Storage when the Advanced view is enabled.

  • Code View for Schemas is now available in the UI. Users can edit (or copy/paste) schemas directly, rather than configuring through UI forms.

[NEW] Performance

  • Cluster overprovisioning support that can be configured to enable faster scaling, by reserving resources.

[NEW] Java SDK

  • Query support has been added to the Java SDK.

  • For ingestion there is an option to validate the data being ingested to ensure it is in the format that is expected by the database, before it is ingested.

  • Batch publishing is available, if required.

[NEW] Observability

  • It is now possible to filter for a displayed value (for example, a particular Container) in the Diagnostics screen. Right-click on the value and select "Filter by [...]".

[NEW] Infrastructure

  • A storage class named sharedfiles, that is backed by shared storage, is automatically created when deploying via client Terraform scripts. sharedfiles storage is used for storing Packages and required by kdb Insights Enterprise.

Fixes

A large number of fixes have gone into kdb Insights Enterprise 1.2.0. Known issues which have been resolved, along with some other highlights, are listed below.

[FIX] Pipelines

  • Pipelines no longer report an initial "Unresponsive" status in the UI when launching.

[FIX] Explore and Query

  • With the introduction of Query Environments, there is no longer an issue where PVCs are not removed upon assembly teardown due to active Explore tabs targetting the assembly.

  • SQL queries can now target splayed tables (reference data).

[FIX] Observability

  • All UI notifications now appear in the Diagnostics panel.

[FIX] UI

  • Redeploying Data (assembly) no longer adds extra nodes to the navigation tree.

Artifacts

type location
Infrastructure kxi-terraform-1.2.0.tgz
Platform insights-1.2.0.tgz
Operator kxi-operator-1.2.0.tgz
CLI kxicli-1.2.0-py3-none-any.whl
ODBC Driver kodbc 1.2.0
Java SDK java-sdk 1.2.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Version 1.2.0 requires a custom storageClassName definition

The addition of support for packages introduces a new storageClassName defined as sharedfiles, this is defined by default in any deployments of kdb Insights Enterprise that rely on the KX provided Terraform scripts for deployment but must be added manually by a user as outlined here.

Alternatively, within the deployment configuration as outlined here, modify the storageClassName to an alternative supported RWX storage option for example rookcephfs.

Keycloak 18

The version of Keycloak used within kdb Insights Enterprise has been upgraded from Keycloak 16 to Keycloak 18.

Please follow the Keycloak 18 migration steps to migrate your Keycloak data prior to upgrading kdb Insights Enterprise.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below 
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data
  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1

  • Torn-down pipelines that include Database or Stream Writers cannot be directly restarted after having their resources removed. Upon redeploy, data will not be published downstream from the pipeline. A workaround is to rename the pipeline. Resources are removed by checking "Clear Pipeline State" during teardown in the UI, or by directly deleting the Kubernetes PVCs backing the pipeline. If either of these things are done, the pipeline must be renamed before relaunching it.

  • .qsp.read.fromDatabase cannot be run in a pipeline to query a kdb Insights Enterprise database. The following example code can be used as a workaround to query data (ex. reference data to be used within a pipeline). 

onResponse:{[hdr;data]
if[0 <> hdr`ac;
    .sp.log.error ("Query failed host=%s, api=%s, code=%n, error=%s";hdr`gw; hdr`api; hdr`ac; hdr`ai)];
publish data
}
.qsp.onStart {
    h:hopen hsym `$.spenv.getReleaseName[],"-sg-gateway:5050";
    res: neg[h] (`.kxi.getData; (enlist `table)!(enlist `instruments); `onResponse; ()!());
}
.qsp.run .qsp.read.fromCallback[`publish] .qsp.write.toConsole[]
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can navigate re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 if you're upgrading from an earlier version, for notes about

1.1.2

kdb Insights Enterprise 1.1.2 is a patch release that corrects a small number of observability and query issues.

Release Date

2022-07-15

Observability

[FIX] Metrics responses are all now terminated with a newline \n. Absence of the newline caused import issues with some metrics scrapers, for example Azure Monitor.

Explore and Queries

[FIX] The getData API endpoint no longer returns an error when an outputTZ is provided.

[FIX] Queries against the interval tier (idb) around the end-of-day rollover process could return zero results.

Infrastructure

[NEW] The Terraform scripts will now pass the correct platform to the Docker container when running them on Macs

[FIX] The Helm version in the Terraform scripts' Dockerfile has been pinned to a version without AWS compatability issues.

Artifacts

type location
Infrastructure kxi-terraform-1.1.1.tgz
Platform insights-1.1.2.tgz
Operator kxi-operator-1.1.0.tgz
CLI kxicli-0.9.0-py3-none-any.whl
ODBC Driver kodbc 1.1.0
Java SDK java-sdk 1.1.0

Upgrade notes

Data loss of UI assembly definitions for versions earlier than 1.1.1

Assembly definitions built from the kdb Insights Enterprise UI in versions earlier than 1.1.1 will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.

To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace. 

kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz

To restore, the user should run the following command post-install. 

cat kxi-controller-data.tgz |  kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -

kdb Insights Operator Version

As the version of the kdb Insights Operator has not changed from 1.1.0 to 1.1.2, the command for deploying kdb Insights Enterprise is slightly different than in earlier releases. The older kdb Insights Operator version must be explicitly passed to the CLI.

kxi install run --filepath values.yaml --version 1.1.2 --operator-version 1.1.0

Please see the guide to Installing kdb Insights Enterprise or Upgrading kdb Insights Enterprise for more detail.

New functionality and roles

kdb Insights Enterprise 1.1.1 includes new features permissioned by additional roles. If you are upgrading to 1.1.2 from a version earlier than 1.1.1, please see the section about authentication upgrades for more information about how to incorporate realm changes, including new roles, during an upgrade.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below 
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.

  • Currently users will be logged out but will not be notified via the UI. If users consistently encounter errors, we suggest they log out and in again.

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • The drop-down list of assemblies in the first Explore tab opened after applying the first kdb Insights Enterprise assembly will take a short while to populate. Try again after a few seconds if nothing appears immediately, or close and re-open the Explore tab to see the list and select an assembly.

  • Teardown cleanups of Data (assemblies) via the UI will not fully remove PVCs if there are active Explore tabs targetting that assembly. Close these tabs or remove their underlying sandboxes via the REST API to complete the cleanup.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 if you're upgrading from an earlier version, for notes about

1.1.1

kdb Insights Enterprise 1.1.1 is a patch release that improves usability and stability.

Release Date

2022-06-28

Data (Assemblies)

[FIX] Clearer tooltips for Data status to ensure users can clearly identify when their assemblies are fully deployed.

[FIX] Additional validation has been added to stop re-deployment of Data (assemblies) after schema changes – schema changes on deployed assemblies are not supported in this version of kdb Insights Enterprise.

[FIX] Additional validation has been added for schemas to ensure they can be deployed successfully.

[FIX] The serviceVolume setting is now mapped correctly in Database and Stream

[FIX] A condition where Stream log archiving could fail has been resolved, ensuring that stream log files do not grow beyond the defined limits.

[FIX] Navigation tree issues where schemas could not be opened and deletion failures were not correctly reflected are both resolved.

Pipelines

[FIX] Nodes requiring passwords no longer store them in the pipeline configuration, they must be entered during pipeline deployment.

[FIX] Mandatory fields in nodes are now enforced in pipelines before deployment.

[FIX] Pipeline re-deployments can now optionally clear state (destroying previously persisted pipeline data, but not data already in the database) to allow edits to be made and the pipeline to be restarted from scratch.

[FIX] The error Cannot read properties of undefined (reading '0') which could be thrown when opening multiple pipeline tabs has been resolved.

[FIX] An issue where long code lines in Map nodes could cause browsers to hang has been resolved.

Explore and Queries

[FIX] Session errors Worker has been closed and r.map are no longer thrown.

[FIX] The error No aggregator available when changing assemblies in an Explore session is no longer thrown.

[FIX] Reference data joins which do not include the foreign column or where the foreign column has a different name are now possible.

[FIX] An issue where certain queries’ WHERE clause would work via REST but fail via Explore has been resolved.

[FIX] An issue where certain queries using count could throw the error Error: noupdate: `.Q `pn has been resolved.

Artifacts

type location
Infrastructure kxi-terraform-1.1.0.tgz
Platform insights-1.1.1.tgz
Operator kxi-operator-1.1.0.tgz
CLI kxicli-0.9.0-py3-none-any.whl
ODBC Driver kodbc 1.1.0
Java SDK java-sdk 1.1.0

Upgrade notes

Data loss of UI assembly definitions

Assembly definitions built from the kdb Insights Enterprise UI will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.

To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace. 

kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz

To restore, the user should run the following command post-install. 

cat kxi-controller-data.tgz |  kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -

kdb Insights Operator Version

As the version of the kdb Insights Operator has not changed from 1.1.0 to 1.1.1, the command for deploying kdb Insights Enterprise is slightly different than in earlier releases. The older kdb Insights Operator version must be explicitly passed to the CLI.

kxi install run --filepath values.yaml --version 1.1.1 --operator-version 1.1.0

Please see the guide to Installing kdb Insights Enterprise or Upgrading kdb Insights Enterprise for more detail.

New functionality and roles

kdb Insights Enterprise 1.1.1 includes new features permissioned by additional roles. Please see the section about authentication upgrades for more information about how to incorporate realm changes, including new roles, during an upgrade.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below 
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.

  • Currently users will be logged out but will not be notified via the UI. If users consistently encounter errors, we suggest they log out and in again.

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • The drop-down list of assemblies in the first Explore tab opened after applying the first Insights assembly will take a short while to populate. Try again after a few seconds if nothing appears immediately, or close and re-open the Explore tab to see the list and select an assembly.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 if you're upgrading from an earlier version, for notes about:

1.1.0

Release for kdb Insights Enterprise.

Release Date

2022-06-07

Features

[NEW] Storage and Query

  • Late and out-of-order data handling
  • Data received out of order is recorded with an appropriate ingest timestamp
  • Late data is recorded, irrespective of where its correct destination is in the lifecycle (RDB, IDB, HDB)
  • Late data is included by default for all requests, q (getData) and ANSI SQL
  • Late data can be filtered out (for example, to see only data present in the system at a particular point in time)
  • Performance is not significantly impacted by late data arrival, including for object storage tiers
  • Reference data can be queried in the Explore Window, in a data pipeline and as part of the getData() API
  • Support for non time-series reference data in Database
  • Data in reference tables can be joined to time-series tables in free-form q queries
  • Improved SQL support for queries (joins and complex queries, except for reference tables)
  • Compression support, including object storage
  • Improved resiliency and performance

[NEW] User Interface

  • Scratchpads
  • Data can be pulled into the scratchpad using either sql or getData()
  • Freeform q, SQL or Python code can be execute on that data to further filter, aggregate and join data to explore specific records
  • Python/q functions or a model can be applied to that data to gain insights

  • A model can be fitted using that data

  • Machine Learning plugins can be added to a Pipeline created from the user interface

  • More intuitive creation of data ingest pipelines
  • Observability embedded into the UI (events and logs tab)
  • Log forwarding to each providers' log aggregator is now enabled by default in the Terraform scripts.

[NEW] Reports

  • Visualization is available for ingested data using the getData() API

[NEW] Observability

  • View Database and Pipeline logs and events from the UI
  • Ability to Export Diagnostics, including the logs, events and state for further diagnosis
  • Further integration with cloud monitoring tools
  • Additional performance metrics collected

[NEW] Reliable Transport Stream

  • Subscribers have the option to override the detail truncation options for the stream logs that they have processed
  • Improved recovery and performance

Artifacts

type location
Infrastructure kxi-terraform-1.1.0.tgz
Platform insights-1.1.0.tgz
Operator kxi-operator-1.1.0.tgz
CLI kxicli-0.9.0-py3-none-any.whl
ODBC Driver kodbc 1.1.0
Java SDK java-sdk 1.1.0

Upgrade notes

Data loss of UI assembly definitions

Assembly definitions built from the Insights UI will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.

To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace. 

kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz

To restore, the user should run the following command post-install. 

cat kxi-controller-data.tgz |  kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below 
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
  • getData does not support connecting reference data column names that do not match. If the name of a column does not match the name of the column it is a foreign key for, then the getData reference join will fail. Example: The trade table has a column sym which is a foreign key to the msym column in the table market. The join will run into an issue since the column names are different.

  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.

  • Passwords used for authenticating to third-party services are currently stored as part of the pipeline configuration. This will be corrected in the next release. It is recommended that instead of using a password directly that a Kubernetes secret is used whenever possible.

  • Currently users will be logged out but will not be notified via the UI. If users consistently encounter errors, we suggest they log out and in again.

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if Insights Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into Insights. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data

Backward Compatibility

  • If you're upgrading from a previous version or the CRD is already installed, helm will not update the definition automatically. See Operator Overview Helm Warning for details on how to rectify this.

License changes

From 1.1.0 KX Insights has moved to consumption-based licensing using a new license file, kx.lic. The user should generate the license themselves using the klic self-service tool. The documentation for this is available here.

After following the klic docs, a new kx.lic should be available for your cluster.

Then you can upload it as a Kubernetes secret using the following command.

kubectl create secret generic kxi-license --from-file=license=kx.lic

Alternatively delete your existing license secret and re-run the CLI install (kxi install ..), choosing the N when prompted if you have an existing license secret. Assuming the license secret is called kxi-license, the command below will delete it.

kubectl delete secret kxi-license

To revert to the pre-1.1.0 behaviour, the following Helm values should be set for the deployment. For details on this file see here.

global:
  license:
    ..
    onDemand: true
    asFile: false

kxi-acc-svc:
  enabled: false

Single pod Storage Manager

By default, when upgrading to 1.1.0 the Storage Manager will no longer create independent containers for the various processes used to manage data persistence. These processes now all run under a single container as child processes. As part of the upgrade, it is optionally encouraged that any assembly data workflows deployed should be upgraded to align any resources with this change.

The change requires the user to remove resources configuration for eoi, eod and dbm sections of an assembly and update the sm.k8sPolicy with a resources configuration for all these processes which will run as a single container within the sm pod.

For example

  sm:
    ..
    k8sPolicy:
      resources:
        requests:
          memory: "14Gi"
          cpu: "2000m"
        limits:
          memory: "14Gi"
          cpu: "4000m"

However if no actions are taken, KX Insights will remain backwards compatible with the existing configuration and will use the sum of all CPU and memory resources under the sm element, using this sum to populate the sm.k8sPolicy.resources section under the sm element.

1.0.0

Release for kdb Insights Enterprise.

Release Date

2022-03-14

Features

[NEW] UI allows storage and reload of data assemblies.

[NEW] Schemas can be created and deleted in the kdb Insights Enterprise user interface.

[NEW] Assemblies can be created, saved and deployed in the kdb Insights Enterprise user interface.

[NEW] Ability to Export Assembly Definitions.

[NEW] Integration with the new License Management system. Due to these changes, existing licenses will need to be re-issued. Please contact your KX representative for more information.

Artifacts

type location
Infrastructure kxi-terraform-1.0.0.tgz
Platform insights-1.0.0.tgz
CLI kxicli-0.4.0-py3-none-any.whl
Operator kxi-operator
ODBC Driver kodbc 1.0.0
Java SDK java-sdk 1.0.0

Known Issues

  • The Service Gateway may repeatably print a NullPointerException under certain conditions. If this should occur the Kubernetes pod that the service gateway process is running on should be bounced.

  • An OOM event may be triggered when repeatedly bombarded with requests that return a large result set. In the event where an OOM occurs the client will not receive a response; the request may be retried and serviced by other gateways.

  • When choosing an assembly in the Explore Window a ‘500 Internal Server Error’ can be returned if no labels are included in the labels section of the assembly definition.

  • In the prospective menu (the customizable navigation toolbar) not all options available in the menu selector are currently supported, we recommend keeping the default menu options. However, all options are included in the Entity Tree shown when the Home menu is selected.

  • When exporting an Assembly definition from the kdb Insights Enterprise UI the file requires some cleaning before it can be imported into another environment.

    • Remove id entry from tables.
    • Remove apiVersion entry from pipelines and sequencers.
    • Remove line breaks from pipeline specs.

  • When saving a pipeline in the UI on AWS or Azure you need to change the Storage Class from “standard” (which is the default for GCP) to a storage class which exists on AWS or Azure. For example, to “gp2" for AWS, or "default" for Azure. This is done in the Persistence tab when saving. Note, the Storage Class is set separately for the controller and worker. Both must be set correctly. Scroll down to set the worker Storage Class too. Otherwise, the pipeline pod and pvc deployment will get stuck in the “pending” state as no “standard” storage class can be found. Alternatively, create a “standard” storage class per the link storage class resource.

  • Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.

  • Not all stream log files written by the Reliable Transport mechanism (RT) are deleted once the stream has been replicated to the subscriber as the RT stream log archival rules are not being applied to all processes. This will cause the disk space of certain volumes associated with the kdb Insights Enterprise cluster to fill up over time. The more data that is sent through the system the faster this will occur. This may result in out of disk space issues at the producer/subscriber and in the raft cluster. We recommend that you monitor the disk usage and manually delete old stream logs regularly to prevent the disks from filling up. A tool that monitors the disk space could be used to alert you when this intervention is required (for example: Google's Cloud Watch on GCP, or Prometheus). See the section RT Logs for locations of the stream log files and how to delete old stream logs that are not longer required.

  • If a publisher is running and either the out directory on any of your 3 nodes in the RT cluster stops growing, or your Stream Processor is not receiving data, you should restart your publisher. To investigate the progress of messages in the RT cluster stream logs and look at the RT application logs see the section RT Logs.

  • If a user leaves trailing spaces after CLI fields, such as secrets, then the trailing spaces are preserved in the values.yaml file leading to errors on deployment. For example, kxi-certificate in below yaml: 
global:
caIssuer:
name: 'kxi-certificate '
secretName: 'kxi-certificate '
  • The CLI error handling is currently limited. If necessary use the underlying kubectl command to see the issue more clearly. For example, to see a yaml syntax error returned by a modified sdk_sample_assembly.yaml file, run: kubectl apply -f sdk_sample_assembly.yaml.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When using the Sample Java program, provided as part of the Java SDK, replace valFloat:float with valFloat:double in the first line of the sample.csv file before running the program.

  • The Controller expects you to call your idb, myidb, under instances if you set mounts to use different keynames than your instances an error occurs.
dap:
  instances:
    idb:
      size: 1
      mountName: myidb
      gwAssembly: insights
  • Enabling .z handler metrics accumulates memory over time and may cause containers to restart if they hit their memory resource limit.

Backward Compatibility

  • If you're upgrading from a previous version or the CRD is already installed, helm will not update the definition automatically. See Operator Overview Helm Warning for details on how to rectify this.

0.9.0

Beta 1 Release for kdb Insights Enterprise.

Release Date

2021-12-09

Features

[NEW] Run Python and q Machine Learning Models.

  • Users can add a q or Python based model to the ML registry.

  • Users can use q to retrieve either a q or a Python based model from the ML registry.

  • Users can deploy a Stream Processor pipeline that includes ML models from the ML Registry.

[NEW] Run Python functions on streaming data. Users can add Python functions into a Stream Processor pipeline to apply transformations (for example: windowing and filtering) to the streaming data.

[NEW] Support for the Kafka Schema Registry (dynamic schema updates are not yet supported).

[NEW] Users have access a single OpenAPI document. This provides documentation for the following endpoints:

  • Service Gateway

  • Client Controller

  • Information Service

  • Stream Processor

All available operations and authentication methods are documented.

[NEW] The User interface supports using ANSI SQL queries on a single database. These queries are made available via a sandbox to ensure the live database is not modified by the queries.

[NEW] Command line interface for:

  • Installation setup.

  • Enrolling and removing clients.

  • Creating and deleting assemblies.

  • Querying data.

[NEW] Login screens include KX themes

Artifacts

type location
infrastructure https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-terraform/0.1.11/kxi-terraform-0.1.11.tgz
installation https://nexus.dl.kx.com/repository/kx-insights-charts/insights-0.9.0.tgz
cli https://nexus.dl.kx.com/repository/kxi/packages/kxicli/0.1.3/kxicli-0.1.3-py3-none-any.whl

Known Issues

  • Creating a pipeline in the UI that contains an Expression node must be done through the Pipeline editor, the Import screen does not allow correct configuration of an Expression node.

  • If one of the nodes in an RT sequencer is killed or restarts, data may no longer flow through that RT. If the issue occurs on north and south sequencer, it may be sufficient to restart the assembly. Otherwise, the cluster needs to be restarted.

  • RT nodes can under certain conditions consume excessive CPU causing the cluster CPU go up to >95%. If the issue occurs open a shell to the affected node(s) and kill all the pull_server processes. They will be automatically restarted and the CPU usage will drop.

  • RT occasionally reports an inotify error. This is a resource exhaustion issue. If the issue occurs on north and south sequencer, it may be sufficient to restart the assembly. Otherwise, the cluster needs to be restarted.

  • Both SQL and q queries work with assemblies. However, when running a query against a html5evalcongroup process in the UI, SQL is not supported and q must be used instead.

  • Programmatic OpenAPI Insights client generation currently fails due to an issue with the Stream Processor.

Backward Compatibility

  • If you're upgrading from a previous version or the CRD is already installed, helm will not update the definition automatically. See Operator Overview Helm Warning for details on how to rectify this.

0.8.0

Initial Alpha Release for kdb Insights Enterprise.

Release Date

2021-10-27

Features

[NEW] Secure login & authentication using Keycloak, enabling enrollment and removal of client publishers.

[NEW] Role Based Access Control with roles for pipeline viewing and editing, access to the user interface and the ability to query data.

[NEW] Licensing is via a base install key.

[NEW] Data ingestion via:

  • Publishing from the RT command line interface.

  • Reading from Kafka.

  • Publishing using direct qIPC.

  • Batch ingestion from S3 buckets.

[NEW] The User interface supports creation of stream processor pipelines; including data ingestion and transformation, and presentation in grids/charts. For this release, pipelines can be published, but drafts are only saved to the user’s browser - not to kdb Insights Enterprise.

[NEW] Data transformations can use q

[NEW] Data transformations can use a limited Python API (built on embedPy)

[NEW] Data can be streamed out via Kafka.

[NEW] Data can be visualized using the insights user interface

[NEW] Data stored in the kdb databases can be accessed via:

  • The user interface using q

  • A REST API

[NEW] Application Metrics can be generated which can be monitored via Prometheus and Grafana.

Artifacts

type location
infrastructure https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-terraform/0.1.7/kxi-terraform-0.1.7.tgz
installation https://nexus.dl.kx.com/repository/kx-insights-charts/insights-0.8.0.tgz

Known Issues

  • On AWS - deployment of infrastructure needs to be targeted at a region that has 3 availability zones - see https://aws.amazon.com/about-aws/global-infrastructure/regions_az/

  • On AWS(us-east-2) - Due to an issue specific to us-east-2 terraform tries to fetch ec2 information by hitting ec2.us-east-2.amazonaws.com normally this url should return a single public ip address but within an ec2 instance on us-east-2 region it returns 3 private ips which are not accessible via https. For this reason the application should not be deployed to us-east-2.

0.10.0

Beta 2 Release for kdb Insights Enterprise.

Release Date

2022-02-04

Features

[NEW] Support for the Kafka Schema Registry has been extended to include dynamic schema updates.

[NEW] Defining ML models can be done using Python.

  • Users can use Python to retrieve either a q or a Python based model from the ML registry.

  • Users can use Python to save either a q or a Python based model from the ML registry.

[NEW] The ANSI SQL support has been extended to include distributed databases. Currently joins are not supported and tables with the same names in different databases MUST have the same columns and properties. As in the previous release these queries are made available via a sandbox to ensure the live database is not modified by the queries.

[NEW] REST API getData function has been enhanced to include:

  • Basic aggregation functions on individual columns (for example: min, max, avg, stdev)
  • Racking and Filling
  • Filtering
  • Time range options, including temporaility

[NEW] Schemas, Streams and Databases can be created and deleted in the kdb Insights Enterprise user interface.

[NEW] KX ODBC driver for data ingestion. Enabling data ingestion by applications that support ODBC.

[NEW] KX Java SDK for data ingestion.

Artifacts

type location
infrastructure kxi-terraform-0.1.12.tgz
installation insights-0.10.0.tgz
cli kxicli-0.2.0-py3-none-any.whl
operator kxi-operator-0.2.0-rc.19.tgz
ODBC driver RPM package kodbc-0.1.52-rhel.x86_64.rpm
ODBC driver Debian kodbc-0.1.52.deb
ODBC driver Windows executable kodbc-0.1.52.exe
ODBC driver Windows msi kodbc-0.1.52.msi
ODBC driver sample program kodbc-0.1.52.zip
Java SDK rt_client-1.0.0rc6.jar
Java SDK sample program rtdemo-1.0.0rc6-all.jar
Java SDK sample csv file sample.csv

Known Issues

  • The sample assembly yaml files provided specify the same subTopic: "data". To start multiple assemblies in parallel, change the subTopic to a unique value, such as subTopic: "odbc-assembly". The corresponding change will also need to be made when enrolling the client. For example: curl https://${INSIGHTS_HOSTNAME}/clientcontroller/enrol -H "Authorization: Bearer " -d '{ "name": "my-client", "topics": { "insert": "odbc-assembly", "query": "requests" }}'

  • The UI may produce pop-up error messages in various scenarios.

  • The UI does not have Reports available.

  • kdb Insights CLI error handling is currently basic. If the CLI throws a stack trace, use the underlying kubectl command to see the issue more clearly. For example, to see a yaml syntax error returned by a modified basic_assembly.yaml file, run: kubectl apply -f basic_assembly.yaml .

  • KX ODBC driver and Java SDK do not work when using Insights installed on an AWS cluster.

Backward Compatibility

  • If you're upgrading from a previous version or the CRD is already installed, helm will not update the definition automatically. See Operator Overview Helm Warning for details on how to rectify this.