Skip to content

kdb Insights Enterprise - Previous Release Notes

This page outlines release notes from past releases. For the most recent release, refer to the latest release page.

1.6.4

Release Date

2023-11-28

kdb Insights Core Version

This version of kdb Insights Enterprise runs kdb Insights Core 4.0.5.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Improved failure recovery for object storage in Storage Manager. This resolves a case where EOI would not complete and would report SM Flushing filesystem with [directory] errors.

  • Resolved a bug in which EOIs were not completing, and NoSuchKey errors occurring after moving data to object storage.

  • Improvements have been made to the refresh token logic for AWS deployments. Previously, token refresh logic was erroneous if the session was not requesting bucket data, rotating every 30 minutes instead of at the intended deadline.

  • Added a fix for the GCP system command expiry timer being discarded/ignored, which resulted in tokens not refreshing at all using gcloud auth.

Security

  • Various CVEs were remediated as part of this release.

Artifacts

type location
Enterprise insights-1.6.4.tgz
Operator kxi-operator-1.6.1.tgz
CLI kxicli-1.6.1-py3-none-any.whl
C SDK kxi-c-sdk 1.6.0
Java SDK kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.6.0
Infrastructure kxi-terraform-1.6.1.tgz

1.6.3

Release Date

2023-07-25

Improvements

  • A new quick test deploy option is available for pipelines for faster iteration during development. This deploys a pipeline in test mode to the user’s scratchpad process and is significantly faster than a full test. See quick test deploy for more information.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Fixed a potential race condition between NFS replication and DAP reload signals. It was previously possible that a DAP may reload before NFS replicated data, which could result in data gaps. A new sequencing mechanism has been added to ensure DAPs do not reload until replication is complete.

  • Fixed an issue where converting a column from a string to a symbol would throw a type error.

  • An error has been fixed that would occur when updating an object storage tier that used an inventory file after a data migration.

  • A new warning has been added to the Resource Coordinator if no Aggregators connect to it. After a wait period, if an Aggregator continues not to connect, this will be escalated to an error to indicate that the system is not ready for queries.

  • REST queries could have input parameters converted to the wrong type when multiple type options were possible. REST APIs were converting to the maximum possible type instead of using the correct default type.

  • Database names in diagnostics now correctly correspond to the name of the assembly that the database maps to.

User Interface

  • Any change made on the Database UI tab will now mark the database as unsaved, and warn the user if trying to close the tab with any unsaved changes.

  • The View Errors button is now renamed to View Database Errors and points to the database error logs. To view Pipeline Errors, click on the Open Pipeline Errors icon next to each pipeline error.

  • Fixed an issue where renaming columns in the schema UI would incorrectly delete the underlying column and create a new one with null data. Column renames in the UI now correctly preserve the column’s data.

Artifacts

type location
Enterprise insights-1.6.3.tgz
Operator kxi-operator-1.6.1.tgz
CLI kxicli-1.6.1-py3-none-any.whl
C SDK kxi-c-sdk 1.6.0
Java SDK kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.6.0
Infrastructure kxi-terraform-1.6.1.tgz

1.6.2

Release Date

2023-07-03

New Features

  • A file watcher for polling cloud storage is now supported in the Stream Processor. The file watcher can monitor any Azure, AWS, or GCP object storage directory. It polls for new files to ingest into an SP pipeline. See the q reader API for details on how to configure the watch parameter in your SP reader. This feature will be available in the Python API and UI in a future release.

  • Query resiliency has been improved by adding the ability to control how processes serving queries are shut down. These processes can now be configured to stay up to serve in-progress query requests before shutting down, ensuring users still receive their data and are unimpacted by system operations. Users simply define how long a process should continue to serve requests before ultimately going offline. This is configured by setting .values.service-gateway.aggregator.aggregator.terminationGracePeriodSeconds to an appropriate duration in your values file during installation.

  • Query scalability has increased with the new ability to independently scale query processes within an assembly, allowing each assembly to have a tailored amount of these processes. Making this configurable and manually scalable lets you better size your assemblies with specific query workloads in mind. See routing configuration for more details. Furthermore, a new routing parameter called scope supports routing query requests based on your target assembly(s).

  • Reading directly from a Parquet file(s) in local storage is now supported in Stream Processor pipelines, allowing users to take advantage of this efficient common file format. See the q reader API for more details. This feature will be available in the Python API and UI in a future release.

  • Connect your existing kdb database already in object storage to your kdb Insights Enterprise system and query it.

  • Pipelines created in the UI can now be exported to be run by the CLI, either for automation or for migrating between environments. From the UI, select the pipeline and click ‘Export’. See Exporting artifacts for more details.

Fixes

Expand to see the full detail of resolved issues here

User Interface

  • In the kdb Insights Enterprise UI, new validation has been added for table names within schemas to ensure that they do not contain invalid characters. Previously, invalid characters could be included in tables names and the database would error after it was deployed.

  • In the kdb Insights Enterprise UI, you can now allow an internal stream to read data from an external source directly to a database. See stream configuration for more details.

  • When creating a new database or renaming an existing database, you are prevented from entering a database name longer than 24 characters.

Scratchpads

  • Fixed an issue in 1.6.0 where evaluating any expressions in the Python editor would throw a type error.

Reliable Transport

  • RT logs are now properly written to a directory defined by the user. Previously, the user's selection was not honoured.

Artifacts

type location
Enterprise insights-1.6.2.tgz
Operator kxi-operator-1.6.1.tgz
CLI kxicli-1.6.1-py3-none-any.whl
C SDK kxi-c-sdk 1.6.0
Java SDK kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.6.0
Infrastructure kxi-terraform-1.6.0.tgz

1.6.1

Release Date

2023-06-22

Improvements

  • In the kdb Insights Database, a new routing parameter called scope has been introduced for routing query requests based on assembly. This parameter allows the user to specifically target individual assemblies for their query.

Artifacts

type location
Enterprise insights-1.6.1.tgz
Operator kxi-operator-1.6.0.tgz
CLI kxicli-1.6.0-py3-none-any.whl
C SDK kxi-c-sdk 1.6.0
Java SDK kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.6.0
Infrastructure kxi-terraform-1.6.0.tgz

1.6.0

Release Date

2023-06-20

Improvements

  • Support for air-gapped environments has been improved and both kxi install run and kxi install upgrade support installing packaged charts (chart tgz’s) by passing the packaged chart name as an argument. If you use the default value of kx-insights for the chart repo, you can install it as follows:

    kxi install run --version 1.6.0 --filepath values.yaml insights-1.6.0.tgz

    This argument also supports chart references, for example:

    kxi install run --version 1.6.0 --filepath values.yaml <CHART_REPO>/insights

  • The kdb Insights Database can now work with or without an inventory file provided for an object storage tier. Using an inventory file for your object storage tier is still the recommended configuration, as this leverages caching and greatly improves performance when querying object storage data.

  • The kdb Insights Enterprise UI has an improved left hand menu. It can be minimized, and it is displayed alongside the Pipeline Operators list when the pipeline editor is active.

  • The kdb Insights Database now validates that primary keys for a partitioned table include the partition column. Previously, setting a primary key on a partitioned table that was not the partition column would error when queried.

  • Query timeout messages now indicate a reason for the timeout. Previously, query timeouts gave no reason, or said “timed out for unspecified reasons”. See queueing timeouts for more details.

  • Greater stability through the use of Kubernetes discovery between the Service Gateway and database processes.

  • Client-side pivot queries are now available in Views, with optional server-side computation for kdb+/q.

  • When calling kxi install delete the kdb Insights CLI now backs up assemblies before tearing them down.

Fixes

Expand to see the full detail of resolved issues here

Security

  • Various CVEs have been remediated as part of this release.

Database

  • Resiliency improvements have been made to ensure the kdb Insights Database can recover from multiple days without end-of-day processing having completed. Previously, if more than 2 EODs were missed, it was possible to get a misaligned error on recovery.

  • Validation has been added for assemblies to ensure that tiers' scheduled alignment time (snap value) are provided as the kdb+ time type (19h). Previously users could enter a timespan which would silently be ignored. Entering a timespan will now error.

Reliable Transport

  • Kubernetes service account tokens are now automatically refreshed from disk. In previous versions these were not refreshed, causing failing requests to the Kubernetes control plane and pods had to be restarted.

Deprecation

CLI Install Commands

The following three arguments have been deprecated from kxi install setup: chart-repo-name, chart-repo-username and chart-repo-url.

To prevent breaking changes the arguments will still work, but they may be removed in a future version. Adding a chart repository is now decoupled from kxi commands. The suggested approach when calling kxi install setup is to add the chart repository directly using helm repo add and to reference that repository by passing the chart reference as an argument.

If you use the default value of kx-insights for the chart repo, you can install it as follows:

kxi install run --version 1.6.0 --filepath values.yaml

If you have used a non-default chart repo value then you can install with:

kxi install run --version 1.6.0 --filepath values <CHART_REPO>/insights

CLI Lifecycle Commands

The following commands have been removed. Please use the equivalents below to manage assemblies.

removed supported command
kxi azure assembly backup kxi assembly backup
kxi azure assembly delete kxi assembly teardown
kxi azure assembly restore kxi assembly deploy

Third-party dependencies

kdb Insights Enterprise 1.6.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.6.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.6.0.tgz
Operator kxi-operator-1.6.0.tgz
CLI kxicli-1.6.0-py3-none-any.whl
C SDK kxi-c-sdk 1.6.0
Java SDK kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.6.0
Infrastructure kxi-terraform-1.6.0.tgz

Known Issues

  • In 1.6.0 only, any code evaluated in the Python editor will throw a type error. To fix the Python editor for this issue you can run the following in a q editor:

    \d .com_kx_edi
python.run: {
    z: python.i.removeExtraIndents z;
    a: python.i.run[z; y];
    $[x;$[y;python.i.truncate (),a;a];$[y;"";::]]
    }

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

    To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below

    klic license renew $LICENSE_UUID --version=latest

env QLIC=$(pwd) klic license fetch $LICENSE_UUID

    Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.

    printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml

kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml
  • When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. Therefore, there is a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.
  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.
  • A known issue exists within the kxi install rollback command when rolling back to prior versions. When the target version was installed with keycloak.initUser enabled. It is expected the keycloak-config-cli job will fail. This is due to the rollback trying to revert that user's password and violating the password history policy.

The rollback failure will manifest as the failure of the keycloak-config-cli job. This will create job pods as it fails and retries. In order to proceed with the rollback it's recommended to delete kubectl delete job ${RELEASE_NAME}-keycloak-config-cli.

A known issue exists during a reinstall using the CLI where the keycloak.initUser is enabled. If the user's password has changed, the reinstall will try to revert it back and be rejected by the password history policy.

This can be avoided by disabling the user import on reinstall using --import-users False as below:

kxi install run --import-users False ..

Backward Compatibility

If performing a rollback from 1.6.0 to 1.5.x, a rollback chart is required. See Upgrading for more details.

1.5.3

Release Date

2023-06-19

Improvements

  • Resiliency improvements have been made to ensure the kdb Insights Database can recover from multiple days without end-of-day processing having completed. Previously, if more than 2 EODs were missed, it was possible to get a misaligned error on recovery.

  • The kdb Insights Database can now work with or without an inventory file provided for an object storage tier. Using an inventory file for your object storage tier is still the recommended configuration, as this leverages caching and greatly improves performance when querying object storage data.

  • In cases where no data transformation is required, the kdb Insights Enterprise UI now allows Streams to be configured to write directly to the database. This reduces the amount of resources required for data ingestion and lowers operating costs.

Fixes

Expand to see the full detail of resolved issues here

Security

  • Various CVEs have been remediated as part of this release.

User Interface

  • When creating a new database or renaming an existing database, you are now prevented from entering a database name longer than 24 characters. Previously, there was no restriction which could lead to name truncation.

  • The Database Schema Code View page now accepts all schema configuration options. Previously, it would not accept blockSize or isSharded.

Reliable Transport

  • Kubernetes service account tokens are now automatically refreshed from disk. In previous versions these were not refreshed, causing failing requests to the Kubernetes control plane and a need to restart your pods to recover.

Artifacts

type location
Enterprise insights-1.5.3.tgz
Operator kxi-operator-1.5.3.tgz
CLI kxicli-1.5.0-py3-none-any.whl
C SDK kxi-c-sdk 1.5.0
Java SDK kxi-java-sdk 1.5.0
RT Bridge rt-bridge 1.5.0
Infrastructure kxi-terraform-1.5.1.tgz

1.5.2

Release Date

2023-05-31

Improvements

  • In the kdb Insights Enterprise UI, when filtering queries in the Query page, distinct is now a supported aggregation type.

  • The getData API can now perform aggregations on foreign key columns.

Fixes

Expand to see the full detail of resolved issues here
  • Kubernetes service account tokens are now automatically refreshed from disk. In previous versions these were not refreshed, causing failing requests to the Kubernetes control plane and pods had to be restarted.

Database

  • When deploying an assembly via the kdb Insights CLI, if ‘size’ is not specified for DAP instances, a default of 3 replicas will once again be applied. In 1.5.1, this default was increased to 4 to provide increased availability, but 3 has been restored as the standard to control costs. Users can configure a greater number of replicas if desired.

  • In the kdb Insights Enterprise UI, pipelines associated with a database now show in the status table. Previously, only the first pipeline would be shown.

  • In the kdb Insights Enterprise UI, the database schema overview now updates when changes are made to the database’s schema. Previously, this would only show the schema state when first viewed.

  • In the kdb Insights Enterprise UI, database documents no longer get marked as modified after a teardown or cleanup.

  • The database configuration UI now validates that the minimum rollover time is 1 minute. Previously, this could be set to 0.

Stream Processor

  • Pipelines can be now be safely torn down without checkpoints having been cleared. Previously, if a teardown was followed by a restart, it is possible the pipeline would never run.

Artifacts

type location
Enterprise insights-1.5.2.tgz //!
Operator kxi-operator-1.5.2.tgz
CLI kxicli-1.5.0-py3-none-any.whl
C SDK kxi-c-sdk 1.5.0
Java SDK kxi-java-sdk 1.5.0
RT Bridge rt-bridge 1.5.0
Infrastructure kxi-terraform-1.5.1.tgz

1.5.0

Release Date

2023-05-09

New Features

UI

  • The kdb Insights Enterprise UI has a new and improved look and user experience for viewing the details of your database(s). All relevant details about a database are now grouped together in a single summary page, giving users an intuitive view into the details and health of its components. Users can quickly see if any errors or warnings are present, and drill down into the details to investigate directly from the page. The details of your schema are conveniently presented for reference as well. To accompany this new interface, the term Assemblies has been simplified to Database in the UI. Assemblies remain the underlying concept tying components together, and are still referenced in documentation. Manual upgrade of old streams and schemas may be required. See upgrading entities in the UI for more details.

Queries

  • To protect against reserved word and column name conflicts, the way labels are passed to a query has changed. Labels should now be specified either as a dictionary, or with the prefix label_. The old label style is still supported in 1.5, but is deprecated and will be disabled by default in 1.6. See upgrade considerations on labels for more details.

  • Distinct has been added to the available aggregations in getData for database queries.

Pipelines

  • Users can now use Python code in all areas of the pipeline UI, allowing for further flexibility in Python user experience with kdb Insights Enterprise. Previously, the addition of Python code to sections of pipeline UI were limited in scope to the function cells of the pipeline configuration forms.

  • The Stream Processor Python API can now leverage automatic schema conversion using a new function kxi.sp.transform.schema.

  • Users can now use the Kafka Schema Registry in Python and the UI. See kxi.sp.read.from_kafka for Python, and the Kafka reader.

  • String manipulation utilities have been added to the UI and the Stream Processor Python and q APIs, allowing users to selectively convert the string data to upper or lowercase representations for the purpose of data cleansing.

  • Non-UI deployments of Stream Processors now support pipeline replicas, providing enhanced resilience to deployed pipelines. Users can now choose to run their pipeline with replicas for high availability in the Stream Processor. This replaces the previous use of pipeline groups.

Reliable Transport

  • Reliable Transport now offers subscribers the ability to use dynamic client-side filtering, enabling users to control what parts of the data feed their subscribers need to receive. This can greatly improve ingestion performance by eliminating the need for subscribers to handle discarding of unnecessary data. See Filtering for more information.

  • RT has introduced logHistory, a new API to assist in analysis and debugging. The API returns metadata about the RT log file history for logs that are no longer available. This includes:

    • The log file name.
    • The time of the first message added to that log file.
    • The position of the first message added to that log file.

    See Log File Meta Data for more information.

Command Line Interface

  • A rollback command has been added to the CLI to simplify rolling back your installed instance of kdb Insights Enterprise after upgrading.

  • A kxi user subcommand has been added to the CLI to support managing users programmatically. This provides greater flexibility than using the keycloak initUser values at deploy time.

Standalone Installation

  • Ability to control which components should be enabled (cert-manager,ingress-nginx, sharedfiles storage class). The available components are defined on the configuration file (kxi-terraform.env).

Improvements

  • Batch ingest now supports ingesting tables that have no symbols. Additionally, extra validation has been added to verify symbol integrity.

  • Batch ingest now supports sorting of columns.

  • The kdb Insights Database gateway now supports returning responses that are greater than 2GB for IPC requests. See routing configuration for more details.

  • Database schemas in the UI can now be modified for assemblies that have been stopped. See schema modification.

  • Requests that supply a log correlator (logCorr) will now see that correlator in the Service Gateway logs. Previously a random correlator was used in the Service Gateway logs.

  • Users can now utilise backtick syntax when defining aggregations in Views, e.g. `c1`avg`price.

  • Pipeline editors can now be resized vertically.

  • Labels for improved accessibility have been added to the Pipelines screen.

  • Added significant improvements for accessibility using keyboard navigation and/or screen readers in the UI.

    • Users can exit a text area in a code block on the Query page by first pressing "CTRL+M" before pressing the "Tab" key.

    • The entity tree menu button can now be opened when hovering your mouse over the entity tree. Arrows can then be used to navigate.

    • "Code View" on the Schemas page can now be closed with the "Esc" key.

    • The Views and Pipeline pages now allow draggable elements to be inserted/dragged onto the page with the use of your keyboard.

- Kubernetes 1.25 is now supported and the default version for standalone installations.

- Ensure the latest AWS CLI is installed on the docker image used for standalone installations

Fixes

Expand to see the full detail of resolved issues here

Database

  • Additional historical tiers that used object storage in the UI now allow dashes and spaces in file names. Previously they would error for dashes and spaces.

  • Migrating tiers to object storage could fail with the eventual error SMDBM Migration discontinued due to failure if multiple threads were used.

  • New validation has been added to prevent accidentally adding a object storage tier to a database that does not already have one. This would previously error at EOD with a file not found error.

  • Schema conversions are now more robust on network based file systems (ex. Rook Ceph). Previously, this could result in a EXDEV. OS Reports: Invalid cross-device link error in the Storage Manager.

  • Changes to disk sort columns in a schema that would be incompatible with the current disk sort will now error with a clear message at deploy time. Previously, this would error with a u-fail error either at an EOI or EOD.

  • Batch ingest can now gracefully handle scenarios where there is no data provided in the session. Previously, this would result in the following error: Error reporting to licensing server: Value must be a long, leaving the ingest pipeline stuck in a RUNNING state.

  • When removing a symbol column from a table, the SM should be able to properly handle its removal as well as the sym file.

  • If an in-memory sort is applied to a column in a schema, it is now sorted on arrival in the RDB. Previously, if data arrived out of order with this set, an s-fail error would be thrown.

  • In the Storage Manager metrics, the kxi_sm_hdb_date_records metric no longer reports an empty value at startup, which resulted in false alerts about the HDB being offline.

  • The Storage Manager now double buffers its state table in order to protect against corruption.

  • Rapid successive queries using qsql, such as those from Views in the kdb Insights Enterprise UI, no longer have the potential to return query results out of order.

  • The kxi_sg_pending metric would previously be decremented twice when a connection closed, resulting in negative pending values.

  • The Service Gateways were creating duplicate connections to Resource Coordinators while they were waiting to come online. This was resulting in a large number of redundant connections being opened between the services. In large deployments, this could result in conn errors in the Resource Coordinator.

  • RDBs and HDBs that require a dependency on the IDB will now error at startup if one is not specified as part of the mount configuration.

User Interface

  • Previously, a database would show a loading spinner if one of its pipelines had reached a ‘finished’ state while its other components were in a ‘ready' state. This has now been rectified and the database will display a ‘ready’ state in this case.

  • An issue with pasting into text fields in a pipeline node’s configuration has been fixed. Previously, the node may be been duplicated.

Stream Processor

  • Pipeline names are now properly displayed in the Running Pipelines section of the Overview page. Previously, these were displaying the pipeline ID and were not clickable for viewing further information.

  • Pipelines deployed via as part of a Database now properly set Kubernetes secrets. Previously, secrets were only being set properly for standalone pipelines.

  • The presence of a UTF-8 byte-order mark at the beginning of CSV files will now be automatically removed from data when acted on by the Stream Processor CSV decoder.

  • Expression readers now return the result of Python lambdas.

  • Window nodes in the Stream Processor can now accept dictionaries as input batches. Previously, incoming dictionaries would be merged with input batch in the buffer, instead of appending to a buffered list of dictionaries.

  • The python write.to_database node previously converted assembly names to booleans. This is now resolved, and also fully supports the direct write functionality.

  • The Stream Processor's Protobuf decoder no longer ignores the value of As List when a pipeline is deployed from the UI. Previously, this value was being overwritten.

  • The Stream Processors Protocol Buffer Encoder node now properly saves the selected Payload Type.

  • The Stream Processor Database Reader now supports SQL statement queries. Previously, this node had erroneously suggested the use of qsql statements for querying which was unsupported.

  • The default onFinish functions for the apply node have been updated to have the correct parameters and examples of flushing a buffer.

  • When using directWrite functionality in .qsp.write.toDatabase, table names as strings are now supported. Previously, this had to be a symbol.

  • Copy/pasting nodes within a pipeline document while other pipeline documents are open in the background no longer produces duplicates. Previously, the copy/paste operation was being applied to both the intended pipeline as well as the unintended pipelines, resulting in nodes disappearing due to shared names/ids.

  • Security vulnerabilities patched in the Stream Processor for the following CVEs: CVE-2022-41910, CVE-2022-41902, CVE-2022-41900, CVE-2022-41880.

  • Query sessions had previously set error trapping to 1 within the q session, resulting in users being unable to interact with their query session. This behaviour manifested in users being unable to interact with their query sessions until they were manually restarted.

Command Line Interface

  • With previous versions of the kdb Insights CLI, the kxi install list-versions command would return an outdated list of versions. This has now been resolved and the list will be updated on each invocation.

Standalone Installation

  • Fix Rook-Ceph OSD not getting assigned if underlying node changes

Third-party dependencies

kdb Insights Enterprise 1.5.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.5.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.5.0.tgz
Operator kxi-operator-1.5.0.tgz
CLI kxicli-1.5.0-py3-none-any.whl
C SDK kxi-c-sdk 1.5.0
Java SDK kxi-java-sdk 1.5.0
RT Bridge rt-bridge 1.5.0
Infrastructure kxi-terraform-1.5.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly from 1.3.x to any newer release, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

Restriction on upgrading to older versions

The CLI will not allow you to "upgrade" to an older version of kdb Insights Enterprise. Please use the rollback command to revert to a previous version.

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

    To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below

    klic license renew $LICENSE_UUID --version=latest

env QLIC=$(pwd) klic license fetch $LICENSE_UUID

    Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.

    printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml

kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml
  • Publishing of a single message larger than 1GB is not supported.
  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
  • When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. Therefore, there is a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

  • A known issue exists within the kxi install rollback command when rolling back to prior versions. When the target version was installed with keycloak.initUser enabled. It is expected the keycloak-config-cli job will fail. This is due to the rollback trying to revert that user's password and violating the password history policy.

The rollback failure will manifest as the failure of the keycloak-config-cli job. This will create job pods as it fails and retries. In order to proceed with the rollback it's recommended to delete kubectl delete job ${RELEASE_NAME}-keycloak-config-cli.

A known issue exists during a reinstall using the CLI where the keycloak.initUser is enabled. If the user's password has changed, the reinstall will try to revert it back and be rejected by the password history policy.

This can be avoided by disabling the user import on reinstall using --import-users False as below:

kxi install run --import-users False ..

Backward Compatibility

If you are upgrading from a version earlier than 1.5.0 you must also read each of the relevant release notes below to ensure you know all the changes that will affect the upgrade.

1.4.5

Release for kdb Insights Enterprise.

Release Date

2023-04-26

Improvements

[IMPROVEMENT]: If publishing a message larger than 1GB, the publisher will now receive a messsage too large error. Users can catch the error (need more clear detail) to avoid getting an error at runtime. This would previously cause downstream subcriber processes to run out of memory during processing.

[IMPROVEMENT]: SQL queries that filter by time but do not explicitly select the time column are now supported. This would previously error.

Fixes

[FIX]: Fixed an issue where rapid successive queries using qsql such as those from Views in the kdb Insights UI could return query results out of order.

[FIX]: Schema conversions are now more robust on network based file systems (ex. Rook Ceph). Previously, this could result in a EXDEV. OS Reports: Invalid cross-device link error in the Storage Manager.

[FIX]: Window nodes in the Stream Processor can now accept dictionaries as input batches. Previously, incoming dictionaries would be merged with input batch in the buffer, instead of appending to a buffered list of dictionaries.

Third-party dependencies

kdb Insights Enterprise 1.4 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.4.5.tgz
Operator kxi-operator-1.4.2.tgz
CLI kxicli-1.4.2-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.1.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly created using a version prior to 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

Licensing

A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below.

klic license renew $LICENSE_UUID --version=latest
env QLIC=$(pwd) klic license fetch $LICENSE_UUID

Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.

printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml
kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Monitoring

We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes. 

If you wish to continue with the higher frequency you can take one of the following steps:

-  Update the interval field within their charts.

    ```yaml
    metrics:
      serviceMonitor:
        interval: "30s"
    ```

- Override the chart values using the global value within insights/values.yaml.

    ```yaml
    global:
      metrics:
        serviceMonitor:
          interval: "30s"
    ```

- Update your assembly yaml, to override the default.

    ```yaml
    spec:
      metrics:
        enabled: true
        serviceMonitor:
          enabled: true
          interval: 10s
    ```

- Update your default within the Assembly Resource - This is generated using the insights/values.yaml.

    ```yaml
    kxi-operator:
      config:
        sidecar:
          metrics:
            serviceMonitor:
              interval: 30s
    ```

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • Publishing of a single message larger than 1GB is not supported.
  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored. 

"[] Failed to compare saved assembly  to deployed version, error: type"

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

  • When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

  • Screen readers will not read out node information represented in a pop-up when hovered over, and may freeze when attempting to read a large amount of content.

  • Unable to navigate up and down cells and read all table content when navigating using only a keyboard.

  • The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml. 
keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.4

Release for kdb Insights Enterprise.

Release Date

2023-04-20

Improvements

[IMPROVEMENT]: The kdb Insights Database now supports a REST interface for loading historical datasets into an existing database. You can now perform a historical batch ingest using a simple REST API. For more details, see batch ingestion.

[IMPROVEMENT]: Query requests that supply a log correlator (logCorr) will now see that correlator in the Service Gateway logs, allowing for easier traceability in query execution and an improved debugging experience. Previously a random correlator was used in the Service Gateway logs.

[IMPROVEMENT]: Added significant improvements for accessibility using keyboard navigation and/or screen readers in the UI.

  • Users can exit a text area in a code block on the Query page by first pressing "CTRL+M" before pressing the "Tab" key.

  • The entity tree menu button can now be opened when hovering your mouse over the entity tree. Arrows can then be used to navigate.

  • "Code View" on the Schemas page can now be closed with the "Esc" key.

  • The Views and Pipeline pages now allow draggable elements to be inserted/dragged onto the page with the use of your keyboard.

Fixes

[FIX]: Security vulnerabilities patched in the Stream Processor for the following CVEs: CVE-2022-41910, CVE-2022-41902, CVE-2022-41900, CVE-2022-41880.

[FIX]: Additional historical tiers that use object storage in the UI now allow dashes and spaces in file names. Previously they would error for dashes and spaces.

[FIX]: Migrating tiers to object storage could fail with the eventual error SMDBM Migration discontinued due to failure if multiple threads were used.

[FIX]: If an SM client cannot connect to SM after a number of retries, it will now restart the pod to try to automatically recover. This resolves connection issues in DAPs when there are network connectivity issues.

[FIX]: Reading of files from Azure Storage larger than ~1MB now successfully writes to a kdb Insights database. Previously this was erroring due to an issue in the chunking of larger files.

[FIX]: The Service Gateway was creating duplicate connections to Resource Coordinators while they were waiting to come online. This was resulting in a large number of redundant connections being opened between the services. In large deployments, this could result in connection errors in the Resource Coordinator.

[FIX]: Expression readers now return the result of Python lambdas.

[FIX]: After upgrading kdb Insights Enterprise from 1.3.x to 1.4.x, clicking a node in a previously-saved pipeline in the UI no longer causes the function definition to flicker and disappear.

[FIX]: Fixed issue with 1.4.0 kxi-terraform scripts, where Terragrunt config was incompatible and was causing deployment failures on AWS.

Third-party dependencies

kdb Insights Enterprise 1.4 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4 release of the standalone infrastructure installation scripts:target="_blank"}.

Artifacts

type location
Enterprise insights-1.4.4.tgz
Operator kxi-operator-1.4.2.tgz
CLI kxicli-1.4.2-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.1.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly created using a version prior to 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

Licensing

A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below.

klic license renew $LICENSE_UUID --version=latest
env QLIC=$(pwd) klic license fetch $LICENSE_UUID

Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.

printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml
kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Monitoring

We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes. 

If you wish to continue with the higher frequency you can take one of the following steps:

-  Update the interval field within their charts.

    ```yaml
    metrics:
      serviceMonitor:
        interval: "30s"
    ```

- Override the chart values using the global value within insights/values.yaml.

    ```yaml
    global:
      metrics:
        serviceMonitor:
          interval: "30s"
    ```

- Update your assembly yaml, to override the default.

    ```yaml
    spec:
      metrics:
        enabled: true
        serviceMonitor:
          enabled: true
          interval: 10s
    ```

- Update your default within the Assembly Resource - This is generated using the insights/values.yaml.

    ```yaml
    kxi-operator:
      config:
        sidecar:
          metrics:
            serviceMonitor:
              interval: 30s
    ```

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.
  • The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored. 
"[] Failed to compare saved assembly  to deployed version, error: type"

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

  • When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

  • Screen readers will not read out node information represented in a pop-up when hovered over, and may freeze when attempting to read a large amount of content.

  • Unable to navigate up and down cells and read all table content when navigating using only a keyboard.

  • The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml. 
keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.3

Release Date

2023-03-31

Improvements

[IMPROVEMENT] We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes. 

If you wish to continue with the higher frequency you can take one of the following steps:

-  Update the interval field within their charts.

    ```yaml
    metrics:
    serviceMonitor:
        interval: "30s"
    Assemblies
    ```

  • Update your assembly yaml, to override the default.

    ```yaml
    spec:
        metrics:
            enabled: true
            serviceMonitor:
            enabled: true
            interval: 10s
```

  • Update your default within the Assembly Resource - This is generated using the insights/values.yaml.

    ```yaml
kxi-operator:
config:
    sidecar:
    metrics:
        serviceMonitor:
        interval: 30s
```

Fixes

[FIX]: Service Gateway: fixed remote code injection vulnerability in Snake YAML CVE-2022-1471

[FIX]: Queries to the kdb Insights Database now have a default timeout of 60s (previously 30s). This allows for longer running queries to be issued from query clients, such as the scratchpad.

[FIX]: Additional historical tiers in the UI now allow for a retention period to be set on the HDB tier. Previously this was erroring due to a validation issue.

[FIX]: In the UI, the real-time database rollover tier now correctly checks that rollover is less than the interval tier.

[FIX]: Requests to object storage (Azure Storage, AWS S3, Google Cloud Storage) will now backoff and retry if they get a throttle request from the vendor. Jitter will be introduced to the retry to ensure that replicated processes do not all retry at the same time.

[FIX]: Setting a column type to Any now correctly uses an anymap type. This was previously reset to an unselected value.

[FIX]: The REST Server's heartbeat logs are now set to TRACE to reduce log output (previously they were logged as INFO logs).

[FIX]: Pasting into a pipeline node’s configuration no longer duplicates the node.

[FIX]: Resolved issue where some pipelines with merge nodes would error with 'missing edge'.

[FIX]: Comments can now be added to the last line of the Scratchpad without causing it to error.

[FIX]: The Scratchpad now has more memory allocated for ad-hoc queries in the kdb Insights Enterprise query panel.

[FIX]: Stream Processor file reader nodes (Azure, AWS S3, File and Google storage) would only process the first file presented even when multiple files were present.

[FIX]: Object storage tiers can now be added to a database via the kdb Insights Enterprise UI.

Third-party dependencies

kdb Insights Enterprise 1.4.3 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.3 release of the standalone infrastructure installation scripts.

Artifacts

type location

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored. 

"[] Failed to compare saved assembly  to deployed version, error: type"

  • Please ensure all pipelines have different names. Two pipelines with the same name may not be deploy correctly.

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml. 

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.2

Release for kdb Insights Enterprise.

Release Date

2023-03-17

Fixes

[FIX] Resource utilization in the UI Scratchpad has been greatly improved. This fix along with a larger default memory allocation provide greater Scratchpad process stability.

Third-party dependencies

kdb Insights Enterprise 1.4.2 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.4.2.tgz
Operator kxi-operator-1.4.0.tgz
CLI kxicli-1.4.1-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored. 

"[] Failed to compare saved assembly  to deployed version, error: type"

  • When creating a pipeline the user interface does not stop you from creating it with the same name as an existing pipeline, however two pipelines with the same name may not be deploy correctly. Please ensure all pipelines have different names.

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml. 

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.1

Release for kdb Insights Enterprise.

Release Date

2023-03-14

Fixes

[FIX] In 1.3.3 and 1.4.0, if the Storage Manager falls behind on EODs for more than a full day, IDB links could be written incorrectly when late data was present. If this issue is encountered, manual data changes are required to recover the database including backup of the stream logs from the date of the failure. Please contact KX Support for help with recovery.

This issue is characterized by a log statement from the Storage Manager as follows: "Unexpected condition: EOXa watermark date YYYY.MM.DD is less than HDB stream date YYYY.MM.DD, setting it to match HDB stream date". The IDB may also start reporting "No such file or directory" on ordinal partitions.

Third-party dependencies

kdb Insights Enterprise 1.4.1 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.4.1.tgz
Operator kxi-operator-1.4.0.tgz
CLI kxicli-1.4.1-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database - this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • If end-of-day processing is interrupted and remains incomplete until the next EOD period, this prevents further EODs from completing and requires manual recovery.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored. 

"[] Failed to compare saved assembly  to deployed version, error: type"

  • Please ensure all pipelines have different names. Two pipelines with the same name may not be deploy correctly.

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml. 

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.0

Release for kdb Insights Enterprise.

Release Date

2023-03-03

New Features

Introducing Pythonic pipeline development in the UI, dynamic logging notifications providing direct access to root cause of underlying issues, an enhanced user experience for database configuration and additional security enhancements.

[NEW] Pipelines

  • You can now use Python in the pipeline builder in the UI to perform aggregations and analytics.

  • Ingest large batches of data directly into the historical database has been moved out of beta.

    • Use fewer resources. Memory and disk resources greatly reduced during ingest.
    • Select the Write Direct to HDB field in the Database Writer in the UI, or use the q and Python APIs.

[NEW] User Interface

  • Improvements to the observability user experience with two new icons in top right hand-side of the UI banner:

    • A notification icon shows a count of new log messages. This ensures you are notified of any new error messages wherever you are in the UI. You can also click on the icon to show more details of the message, these are displayed in the right-hand third of the screen.

    • A settings icon enables you to update the application logging level for future log records written by the kdb Insights Enterprise components.

  • You can now share the URL of a View by selecting the view on the left-hand toolbar, right-clicking and choosing the new Share button. This URL just shares the content of the View and not the toolbars (e.g. it does not show the left-hand toolbar or the Design/Preview panel at the top).

[NEW] Databases

A new version of our Database Configuration page is now available, under a beta flag, that provides an enhanced user experience. It can be toggled on and off from the Settings option in the main toolbar. We recommend that you use the existing Database Configuration age for all running databases, but please try the new user interface out and give us your feedback.

[NEW] Packages

The user interface now allows you to configure which Packages and their associated Custom APIs are used within individual databases, this had previously only been possible through kdb CLI deployments.

[NEW] Security Improvements

  • Keycloak has been updated to version 19. This impacts the following:
    • The Keycloak Admin console has been changed
    • Logout screen provides a redirection to the login page

Improvements

Keycloak realm

In this release, the handling of the Keycloak realm has been improved. This realm contains the role definitions, users and clients. Previously the realm was not being imported by default on install. The role definitions will always be imported as part of an install or upgrade now. The users and clients will only be imported by default on install (kxi install), and not on upgrade.

To make changes to the initClient and initUser on upgrade, you will need to tell the application to import them using the setting below.

keycloak:
  importUsers: true

Reliable Transport

The kdb Insights Enterprise transport layer has adopted a version of Raft called NuRaft. This improves the fault tolerance and performance of kdb Insights Enterprise. NuRaft is an open source version of Raft. Migrating to NuRaft also provides kdb Insights Enterprise with additional capabilities for future enhancements to Reliable Transport.

Third-party dependencies

kdb Insights Enterprise 1.4.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts .

Artifacts

type location
Enterprise insights-1.4.0.tgz
Operator kxi-operator-1.4.0.tgz
CLI kxicli-1.4.0-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database - this will reduce (but not eliminate) the amount of duplicate data

  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml

  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
Tearing down assembly dfx-assembly
Are you sure you want to teardown dfx-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%
$ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
Tearing down assembly iot-assembly
Are you sure you want to teardown iot-assembly [y/N]: y
Waiting for assembly to be torn down  [------------------------------------]    0%

  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>

  6. Carry out the upgrade

  7. Restore assemblies

    $ kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps .

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Supported Python versions in the CLI

Support for Python 3.7 has been dropped in the CLI.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • If end-of-day processing is interrupted and remains incomplete until the next EOD period, this prevents further EODs from completing and requires manual recovery.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored. 

"[] Failed to compare saved assembly  to deployed version, error: type"

  • When creating a pipeline the user interface does not stop you from creating it with the same name as an existing pipeline, however two pipelines with the same name may not be deploy correctly. Please ensure all pipelines have different names.

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml. 

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.3.4

Release for kdb Insights Enterprise.

Release Date

2023-03-13

Fixes

[FIX] As of 1.3.3, if the Storage Manager falls behind on EODs for more than a full day, IDB links could be written incorrectly when late data was present. If this issue is encountered, manual data changes are required to recover the database including backup of the stream logs from the date of the failure. Please contact KX Support for help with recovery.

This issue is characterized by a log statement from the Storage Manager as follows: "Unexpected condition: EOXa watermark date YYYY.MM.DD is less than HDB stream date YYYY.MM.DD, setting it to match HDB stream date". The IDB may also start reporting "No such file or directory" on ordinal partitions.

Artifacts

type location
Infrastructure kxi-terraform-1.3.1.tgz
kdb Insights Enterprise insights-1.3.4.tgz
Operator kxi-operator-1.3.1.tgz
CLI kxicli-1.3.1-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0

Batch Ingest Known Issues

  • In the unlikely event of two batch ingests being done within an end of interval (EOI) event (for example, a 10 min window) a one hour delay would occur for the subsequent ingest, but there would not be any data loss, just a delay.
  • If the Storage Manager (SM) crashes during a batch ingest, the ingest will fail. On SM recovery, it will not try again and must be restarted, but no clean-up is required.

Upgrade Notes

  • When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Resource Coordinator

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" can be misleading in the timeout message. It should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

Python Scratchpad for Queries

  • If the script consists of multiple statements, the console does not print the output of the last evaluation, and instead it only displays the expression before it is evaluated.

S3 Import Wizard

  • If you select the source, S3, and click Next, no error will be thrown, which is invalid. At least one file needs to be present.

Keycloak Config CLI

We recommend that you enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default Password Policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the Insights values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

If you currently set initUser.enabled=true or initClient.enabled=true within your own values.yaml, you may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where a Keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, you must set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly blockSize Changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

  • if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
  • this has the highest performance, but has no RAM limit on received data
  • if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
  • the smaller this number is set to, the worse ingest performance but stronger RAM limits; this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1
  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation does not rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, you will not be redirected back to the login screen.
  • If you were logged out due to inactivity, you might see a "Logout failed" error.

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}.

1.3.3

Release for kdb Insights Enterprise.

Release date

2023-01-17

New features

Functionality for improved ingestion, efficiency and reduction of resources. Take advantage of an IDE to develop your databases, pipelines and query interfaces.

[NEW] Improved ingestion

  • [Beta] Ingest large batches of data directly into the historical database:

    • Use fewer resources. Memory and disk resources greatly reduced during ingest.
    • Select the Direct Writing field in the Database Writer in the UI, or use the q and Python APIs.

    See details here for known issues.

[NEW] Writing data to Amazon S3

  • You can now write data to Amazon S3 by either:

[NEW] Improved IDE capabilities

  • Build YAML files and upload them using kdb Insights CLI:

    • Speed up development workflow, improve productivity.
    • Access to the Kubernetes control plane is no longer needed. This is now handled for you by the CLI.
    • Assembly management using CLI.

  • If you have access to KX Analyst, you can use it to develop functions. You can now:

    • Enjoy a rich IDE development environment with syntax highlighting for YAML files.
    • Query kdb Insights Enterprise data using REST API or query interface.
    • Upload using the CLI or as part of CI/CD pipeline.
    • Build a complete SDLC that harnesses the flexibility of KX Analyst against kdb Insights Enterprise data.

    Note

    Contact your KX Sales representative if you are interested in using KX Analyst. In future releases, additional IDEs will become available, including VS Code and Jupyter Notebooks.

Improvements

[NEW] Reliable Transport

  • Reliable Transport's logging default level for the sequencer and merger has been changed from DEBUG to INFO.

Fixes

[FIX] Following a hard reset of the Reliable Transport (RT), the log files in the out folder are truncated rather than deleted. This means that the replicated log files for all subscribers to the RT instance will be garbage collected once the RT archiver starts after the reset. This prevents a hard reset from causing subscriber PVCs to fill up due to old/corrupt log files remaining.

Artifacts

type location
Infrastructure kxi-terraform-1.3.1.tgz
kdb Insights Enterprise insights-1.3.3.tgz
Operator kxi-operator-1.3.1.tgz
CLI kxicli-1.3.1-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0
RT Bridge rt-bridge-1.4.0-install.sh

Known Issues

Batch ingest

  • In the unlikely event of two batch ingests being done within an end of interval (EOI) event (for example, a 10 min window) a one hour delay would occur for the subsequent ingest, but there would not be any data loss, just a delay.
  • If the Storage Manager (SM) crashes during a batch ingest, the ingest will fail. On SM recovery, it will not try again and must be restarted, but no clean-up is required.

Discovery Service

  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.

Upgrade Notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Terraform Script

Note the version. GKE cluster cannot be deployed using a 1.3.0 version of the script; a 1.3.* version of the client scripts is needed to use Kubernetes v1.22 on GKE.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the Keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

If you currently set initUser.enabled=true or initClient.enabled=true within your values.yaml, you may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where Keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly blockSize changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

  • If unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval. This has the highest performance, but has no RAM limit on received data
  • If set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM. The smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed.

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

If the script consists of multiple statements, the console does not print the output of the last evaluation, and instead it only displays the expression before it is evaluated.

If S3 is selected as the source and Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

1.3.2

Release for kdb Insights Enterprise.

Release Date

2022-12-05

New Features

[NEW] Pipelines

  • [Beta] Compressed files (GZIP) can now be decoded using the q and Python APIs.

  • Multiple file paths can be added to a cloud storage Reader node using the + button in the UI, or via the [q]https://code.kx.com/insights/1.3/api/stream-processor/q/readers.html){:target="_blank"} and Python APIs.

  • Wildcard support using * is now available when using a single file path with the Amazon S3 Reader.

  • CSV encoding is now available, using a new node in the UI, or via the q and Python APIs.

  • It is now possible to write to Amazon S3 buckets, using a new node in the UI, or via the q and Python APIs.

  • Pipeline lifecycle events can be subscribed to via the q API. This can be used, for example, to invoke a callback once files have been processed by a pipeline.

Improvements

  • Queries will no longer time out after 60 seconds at the gateway. Longer timeouts can be configured in getData API calls using the timeout header option.

Fixes

[FIX] Query

  • An issue where DAPs could enter low-memory mode earlier than they should has been resolved.

Artifacts

type location
Infrastructure kxi-terraform-1.3.0.tgz
Platform insights-1.3.2.tgz
Operator kxi-operator-1.3.1.tgz
CLI kxicli-1.3.0-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within kdb Insights Enterprise values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

Users who currently set initUser.enabled=true or initClient.enabled=true within their own values.yaml may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
    .Values.keycloak.initUser.name
    .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
    .Values.keycloak.initClient.clientId
    .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"
Where keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, the user is required to set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly blockSize changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

  • if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
    • this has the highest performance, but has no RAM limit on received data
  • if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
    • the smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1
  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.3.1

Release for kdb Insights Enterprise.

Release Date

2022-11-23

Security

[FIX] kdb Insights Enterprise 1.3.1 contains security updates for a number of critical CVEs

Streams and Reliable Transport

[FIX] An issue which could cause RT archiving to fail has been fixed.

Artifacts

type location
Infrastructure kxi-terraform-1.3.0.tgz
Platform insights-1.3.1.tgz
Operator kxi-operator-1.3.1.tgz
CLI kxicli-1.3.0-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights Enterprise values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

Users who currently set initUser.enabled=true or initClient.enabled=true within their own values.yaml may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
    .Values.keycloak.initUser.name
    .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
    .Values.keycloak.initClient.clientId
    .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"
Where keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, the user is required to set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly blockSize changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

  • if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
    • this has the highest performance, but has no RAM limit on received data
  • if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
    • the smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1
  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.3.0

Release for kdb Insights Enterprise.

Release Date

2022-10-26

Features

[NEW] User Interface

  • Items in the page tree have been renamed for clarity and consistency with the CLI.
  • Data is now known as Assemblies
  • Explore is now known as Queries
  • Reports is now known as Views

[NEW] Pipelines

  • An "Auto" setting for Parse Strings in the Apply Schema node has been added. The column will be parsed if the input is textual (strings or bytes), but the output type is non-textual.

  • Data types are now visible for each column for Test Deploy previews. These can be used to validate the data types and the data shape of Transforms and Maps results.

  • Data types and column names are validated for Database writers as part of a test deploy.

  • Additional Reader validation has been introduced to pipelines, to ensure correctness of settings before deploying.

  • Test deploy previews can now show non-tabular data.

  • Previews are now available for Reader nodes, before data has been decoded.

  • The visibility of a Pipeline's state has been improved in the Overview tab. Warning and error counts can be clicked on to launch a diagnostics tab for that Pipeline.

[NEW] Assemblies

  • It is now possible to see assemblies created using kxi assembly create in the UI. Note that assemblies created and managed through the command-line are not editable via the UI.

  • Additional logging has been added to the Storage Manager, to ease troubleshooting.

  • Schema configuration validation has been improved.

[NEW] Install and Packages

  • The kdb Insights Operator is no longer uninstalled by default by kxi install delete, the command line parameter --uninstall-operator must now be used.

  • Many values for kxi install setup can now be passed via command line parameter, please see the CLI Reference for more detail.

  • Improved support for rolling back to previous versions. Enter a previous version number using the upgrade command kxi install upgrade to roll back.

  • The current kubectl context is now used as the default in all commands, instead of the namespace specified in the cli-config file.

  • kdb Insights Python modules for packaging and machine learning are no longer dependent on libc.musl, and can now be used for development on Windows and Mac.

  • Custom user query APIs and Aggregation functions can be installed and initialised within Data Access Processes and Aggregators as outlined here.

  • Custom user query APIs and Aggregation functions can be installed and initialised within Data Access Processes and Aggregators as outlined here.

[NEW] Observability

  • Reliable Transport diagnostics are now available in diagnostics exports from the UI.

[NEW] Infrastructure

  • The base image for most components has been moved to Rocky Linux.

Fixes

A large number of fixes have gone into kdb Insights Enterprise 1.3.0. Known issues which have been resolved, along with some other highlights, are listed below.

[FIX] Streams and Reliable Transport

  • A hard reset can now be performed on Reliable Transport, to recover from stream corruption. Please contact KX Support for usage instructions.

[FIX] Pipelines

  • Pipelines which have their data cleared when they're torn-down can now be re-deployed with the same name. Previously, this would cause RT sequence number problems, stopping data from flowing.

  • Persistent pop-up errors in the UI after resizing a Map node have been resolved.

  • .qsp.read.fromDatabase can now be used in pipelines.

  • Renaming a pipeline node no longer disconnects its edges.

  • Empty pipelines are no longer deployable.

[FIX] User Interface

  • Validation fixes so that date and time fields can be entered without use of the picker.

[FIX] Java SDK

  • Occasional authentication failures when querying via Java SDK have been resolved. These would previously be seen more frequently on slower connections.

Artifacts

type location
Infrastructure kxi-terraform-1.3.0.tgz
Platform insights-1.3.0.tgz
Operator kxi-operator-1.3.0.tgz
CLI kxicli-1.3.0-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of KX Insights.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights Enterprise values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

Users who currently set initUser.enabled=true or initClient.enabled=true within their own values.yaml may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
    .Values.keycloak.initUser.name
    .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
    .Values.keycloak.initClient.clientId
    .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"
Where keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, the user is required to set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below 
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

  • The user-defined variables & functions in the Queries tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • The Queries tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Queries tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1
  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.2.1

Release for kdb Insights Enterprise.

Release Date

2022-09-22

Fixes

kdb Insights Enterprise 1.2.1 is a security update for vulnerabilities in the included images for Keycloak/PostgreSQL.

Artifacts

type location
Infrastructure kxi-terraform-1.2.0.tgz
Platform insights-1.2.1.tgz
Operator kxi-operator-1.2.0.tgz
CLI kxicli-1.2.0-py3-none-any.whl
ODBC Driver kodbc 1.2.0
Java SDK java-sdk 1.2.0
keycloak-server keycloak-server-0.2.1.tgz

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Version 1.2.1 requires a custom storageClassName definition

The addition of support for packages introduces a new storageClassName defined as sharedfiles, this is defined by default in any deployments of kdb Insights Enterprise that rely on the KX provided Terraform scripts for deployment but must be added manually by a user as outlined here.

Alternatively, within the deployment configuration as outlined here, modify the storageClassName to an alternative supported RWX storage option for example rookcephfs.

Keycloak 18

The version of Keycloak used within kdb Insights Enterprise 1.2.0 and 1.2.1 has been upgraded from Keycloak 16 to Keycloak 18.

Please follow the Keycloak 18 migration steps to migrate your Keycloak data prior to upgrading kdb Insights Enterprise from versions older than 1.2.0.

Azure Marketplace upgrade

Upgrading from v1.2.0 Marketplace release has a known issue. Keycloak was upgraded in v1.2.1 which contained non backward compatible changes. The PostgreSQL secret format has changed, and existingSecret from values.yaml has an extra nested key. Please make sure values.yaml has the extra auth nested key, and that the secret containing PostgreSQL passwords, usually called kxi-postgresql, has dropped the postgresql prefix from the data keys.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below 
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data
  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1

  • Torn-down pipelines that include Database or Stream Writers cannot be directly restarted after having their resources removed. Upon redeploy, data will not be published downstream from the pipeline. A workaround is to rename the pipeline. Resources are removed by checking "Clear Pipeline State" during teardown in the UI, or by directly deleting the Kubernetes PVCs backing the pipeline. If either of these things are done, the pipeline must be renamed before relaunching it.

  • .qsp.read.fromDatabase cannot be run in a pipeline to query a kdb Insights Enterprise database. The following example code can be used as a workaround to query data (ex. reference data to be used within a pipeline). 

onResponse:{[hdr;data]
if[0 <> hdr`ac;
    .sp.log.error ("Query failed host=%s, api=%s, code=%n, error=%s";hdr`gw; hdr`api; hdr`ac; hdr`ai)];
publish data
}
.qsp.onStart {
    h:hopen hsym `$.spenv.getReleaseName[],"-sg-gateway:5050";
    res: neg[h] (`.kxi.getData; (enlist `table)!(enlist `instruments); `onResponse; ()!());
}
.qsp.run .qsp.read.fromCallback[`publish] .qsp.write.toConsole[]
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can navigate re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.2.0

Release for kdb Insights Enterprise.

Release Date

2022-09-02

Features

[NEW] Install and Packages

  • Upgrade support{:target="blank"} in the kxi command line tool for easier upgrades. _Please take note of other Upgrade Notes below.

  • You must now provide a namespace when deleting installs using kxi install delete in the CLI.

  • Ability to list, add and delete custom versioned user code via packages to kdb Insights Enterprise via the CLI outlined here.

  • Support within packages for the definition of User Defined Functions (UDFs) written in Python or q for deployment to stream-processor or use in the Explore Window.

[NEW] Pipelines

  • MQTT QoS 0 Reader has been added.

  • Users can run Test deploys in the UI, which provide user feedback on issues when building or editing a pipeline. This includes visualization of the data at each node in the pipeline, and clear indications of which nodes failed.

  • CPU and Memory resources can now be configured from the User Interface to allow rightsizing of data pipelines.

  • Use of User Defined Functions as part of a pipeline.

  • Apply Schema is now available for use after the CSV decoder, so the optional schema parameters for the CSV node have been removed from the Pipelines UI. Users should take care to ensure they're parsing all fields in the Apply Schema node when it's being applied to CSV decoded data.

[NEW] Explore and Query

  • Significant speed improvements to Explore. Query environments are pre-allocated as part of Data/assemblies, and so switching between tabs, assemblies, or instances is now instant. Query environments must be enabled in the Database/Data Access screen, or in assembly configuration, for the assembly to be available in Explore.

  • Filtering and aggregation are now available on the Query tab.

  • There is no need to choose a database when using the Query or SQL options in the Explore window. Label filtering can be used to select particular databases. You can now read from tables with the same name and schema across multiple databases when using the Query or SQL tabs.

  • Import and use of User Defined Functions in pipelines and the Explore Window to allow running of custom code.

  • Console output can now be cleared by right-clicking in the Console and choosing "Clear".

  • The query section at the top of Explore tabs can now be collapsed for a larger code area.

[NEW] Databases

  • CPU and Memory resources can now be configured from the User Interface to allow rightsizing of storage and query services.

  • Redundant replicas possible for all query-path processes, enabling greater query uptime despite process, node, or availability-zone failure.

  • Ability for Data Access tiers (RDB, IDB, HDB) to share compute resources by using mountList within Assembly configuration rather than mountName.

  • Granular assembly deployment status showing the state of deploying components, and assembly deployment errors, are now surfaced in the UI. Mouse-over the assembly's status icon while it's deploying to see the breakdown.

  • Environmental variables can now be configured in the UI for Data Access and Storage when the Advanced view is enabled.

  • Code View for Schemas is now available in the UI. Users can edit (or copy/paste) schemas directly, rather than configuring through UI forms.

[NEW] Performance

  • Cluster overprovisioning support that can be configured to enable faster scaling, by reserving resources.

[NEW] Java SDK

  • Query support has been added to the Java SDK.

  • For ingestion there is an option to validate the data being ingested to ensure it is in the format that is expected by the database, before it is ingested.

  • Batch publishing is available, if required.

[NEW] Observability

  • It is now possible to filter for a displayed value (for example, a particular Container) in the Diagnostics screen. Right-click on the value and select "Filter by [...]".

[NEW] Infrastructure

  • A storage class named sharedfiles, that is backed by shared storage, is automatically created when deploying via client Terraform scripts. sharedfiles storage is used for storing Packages and required by kdb Insights Enterprise.

Fixes

A large number of fixes have gone into kdb Insights Enterprise 1.2.0. Known issues which have been resolved, along with some other highlights, are listed below.

[FIX] Pipelines

  • Pipelines no longer report an initial "Unresponsive" status in the UI when launching.

[FIX] Explore and Query

  • With the introduction of Query Environments, there is no longer an issue where PVCs are not removed upon assembly teardown due to active Explore tabs targetting the assembly.

  • SQL queries can now target splayed tables (reference data).

[FIX] Observability

  • All UI notifications now appear in the Diagnostics panel.

[FIX] UI

  • Redeploying Data (assembly) no longer adds extra nodes to the navigation tree.

Artifacts

type location
Infrastructure kxi-terraform-1.2.0.tgz
Platform insights-1.2.0.tgz
Operator kxi-operator-1.2.0.tgz
CLI kxicli-1.2.0-py3-none-any.whl
ODBC Driver kodbc 1.2.0
Java SDK java-sdk 1.2.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Version 1.2.0 requires a custom storageClassName definition

The addition of support for packages introduces a new storageClassName defined as sharedfiles, this is defined by default in any deployments of kdb Insights Enterprise that rely on the KX provided Terraform scripts for deployment but must be added manually by a user as outlined here.

Alternatively, within the deployment configuration as outlined here, modify the storageClassName to an alternative supported RWX storage option for example rookcephfs.

Keycloak 18

The version of Keycloak used within kdb Insights Enterprise has been upgraded from Keycloak 16 to Keycloak 18.

Please follow the Keycloak 18 migration steps to migrate your Keycloak data prior to upgrading kdb Insights Enterprise.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below 
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data
  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1. 
sm:
  size: 1

  • Torn-down pipelines that include Database or Stream Writers cannot be directly restarted after having their resources removed. Upon redeploy, data will not be published downstream from the pipeline. A workaround is to rename the pipeline. Resources are removed by checking "Clear Pipeline State" during teardown in the UI, or by directly deleting the Kubernetes PVCs backing the pipeline. If either of these things are done, the pipeline must be renamed before relaunching it.

  • .qsp.read.fromDatabase cannot be run in a pipeline to query a kdb Insights Enterprise database. The following example code can be used as a workaround to query data (ex. reference data to be used within a pipeline). 

onResponse:{[hdr;data]
if[0 <> hdr`ac;
    .sp.log.error ("Query failed host=%s, api=%s, code=%n, error=%s";hdr`gw; hdr`api; hdr`ac; hdr`ai)];
publish data
}
.qsp.onStart {
    h:hopen hsym `$.spenv.getReleaseName[],"-sg-gateway:5050";
    res: neg[h] (`.kxi.getData; (enlist `table)!(enlist `instruments); `onResponse; ()!());
}
.qsp.run .qsp.read.fromCallback[`publish] .qsp.write.toConsole[]
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can navigate re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.1.2

kdb Insights Enterprise 1.1.2 is a patch release that corrects a small number of observability and query issues.

Release Date

2022-07-15

Observability

[FIX] Metrics responses are all now terminated with a newline \n. Absence of the newline caused import issues with some metrics scrapers, for example Azure Monitor.

Explore and Queries

[FIX] The getData API endpoint no longer returns an error when an outputTZ is provided.

[FIX] Queries against the interval tier (idb) around the end-of-day rollover process could return zero results.

Infrastructure

[NEW] The Terraform scripts will now pass the correct platform to the Docker container when running them on Macs

[FIX] The Helm version in the Terraform scripts' Dockerfile has been pinned to a version without AWS compatability issues.

Artifacts

type location
Infrastructure kxi-terraform-1.1.1.tgz
Platform insights-1.1.2.tgz
Operator kxi-operator-1.1.0.tgz
CLI kxicli-0.9.0-py3-none-any.whl
ODBC Driver kodbc 1.1.0
Java SDK java-sdk 1.1.0

Upgrade notes

Data loss of UI assembly definitions for versions earlier than 1.1.1

Assembly definitions built from the kdb Insights Enterprise UI in versions earlier than 1.1.1 will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.

To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace. 

kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz

To restore, the user should run the following command post-install. 

cat kxi-controller-data.tgz |  kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -

kdb Insights Operator Version

As the version of the kdb Insights Operator has not changed from 1.1.0 to 1.1.2, the command for deploying kdb Insights Enterprise is slightly different than in earlier releases. The older kdb Insights Operator version must be explicitly passed to the CLI.

kxi install run --filepath values.yaml --version 1.1.2 --operator-version 1.1.0

Please see the guide to Installing kdb Insights Enterprise or Upgrading kdb Insights Enterprise for more detail.

New functionality and roles

kdb Insights Enterprise 1.1.1 includes new features permissioned by additional roles. If you are upgrading to 1.1.2 from a version earlier than 1.1.1, please see the section about authentication upgrades for more information about how to incorporate realm changes, including new roles, during an upgrade.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below 
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.

  • Currently users will be logged out but will not be notified via the UI. If users consistently encounter errors, we suggest they log out and in again.

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • The drop-down list of assemblies in the first Explore tab opened after applying the first kdb Insights Enterprise assembly will take a short while to populate. Try again after a few seconds if nothing appears immediately, or close and re-open the Explore tab to see the list and select an assembly.

  • Teardown cleanups of Data (assemblies) via the UI will not fully remove PVCs if there are active Explore tabs targetting that assembly. Close these tabs or remove their underlying sandboxes via the REST API to complete the cleanup.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data

1.1.1

kdb Insights Enterprise 1.1.1 is a patch release that improves usability and stability.

Release Date

2022-06-28

Data (Assemblies)

[FIX] Clearer tooltips for Data status to ensure users can clearly identify when their assemblies are fully deployed.

[FIX] Additional validation has been added to stop re-deployment of Data (assemblies) after schema changes – schema changes on deployed assemblies are not supported in this version of kdb Insights Enterprise.

[FIX] Additional validation has been added for schemas to ensure they can be deployed successfully.

[FIX] The serviceVolume setting is now mapped correctly in Database and Stream

[FIX] A condition where Stream log archiving could fail has been resolved, ensuring that stream log files do not grow beyond the defined limits.

[FIX] Navigation tree issues where schemas could not be opened and deletion failures were not correctly reflected are both resolved.

Pipelines

[FIX] Nodes requiring passwords no longer store them in the pipeline configuration, they must be entered during pipeline deployment.

[FIX] Mandatory fields in nodes are now enforced in pipelines before deployment.

[FIX] Pipeline re-deployments can now optionally clear state (destroying previously persisted pipeline data, but not data already in the database) to allow edits to be made and the pipeline to be restarted from scratch.

[FIX] The error Cannot read properties of undefined (reading '0') which could be thrown when opening multiple pipeline tabs has been resolved.

[FIX] An issue where long code lines in Map nodes could cause browsers to hang has been resolved.

Explore and Queries

[FIX] Session errors Worker has been closed and r.map are no longer thrown.

[FIX] The error No aggregator available when changing assemblies in an Explore session is no longer thrown.

[FIX] Reference data joins which do not include the foreign column or where the foreign column has a different name are now possible.

[FIX] An issue where certain queries’ WHERE clause would work via REST but fail via Explore has been resolved.

[FIX] An issue where certain queries using count could throw the error Error: noupdate: `.Q `pn has been resolved.

Artifacts

type location
Infrastructure kxi-terraform-1.1.0.tgz
Platform insights-1.1.1.tgz
Operator kxi-operator-1.1.0.tgz
CLI kxicli-0.9.0-py3-none-any.whl
ODBC Driver kodbc 1.1.0
Java SDK java-sdk 1.1.0

Upgrade notes

Data loss of UI assembly definitions

Assembly definitions built from the kdb Insights Enterprise UI will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.

To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace. 

kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz

To restore, the user should run the following command post-install. 

cat kxi-controller-data.tgz |  kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -

kdb Insights Operator Version

As the version of the kdb Insights Operator has not changed from 1.1.0 to 1.1.1, the command for deploying kdb Insights Enterprise is slightly different than in earlier releases. The older kdb Insights Operator version must be explicitly passed to the CLI.

kxi install run --filepath values.yaml --version 1.1.1 --operator-version 1.1.0

Please see the guide to Installing kdb Insights Enterprise or Upgrading kdb Insights Enterprise for more detail.

New functionality and roles

kdb Insights Enterprise 1.1.1 includes new features permissioned by additional roles. Please see the section about authentication upgrades for more information about how to incorporate realm changes, including new roles, during an upgrade.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below 
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}

  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.

  • Currently users will be logged out but will not be notified via the UI. If users consistently encounter errors, we suggest they log out and in again.

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • The drop-down list of assemblies in the first Explore tab opened after applying the first Insights assembly will take a short while to populate. Try again after a few seconds if nothing appears immediately, or close and re-open the Explore tab to see the list and select an assembly.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 if you're upgrading from an earlier version, for notes about:

1.1.0

Release for kdb Insights Enterprise.

Release Date

2022-06-07

Features

[NEW] Storage and Query

  • Late and out-of-order data handling
  • Data received out of order is recorded with an appropriate ingest timestamp
  • Late data is recorded, irrespective of where its correct destination is in the lifecycle (RDB, IDB, HDB)
  • Late data is included by default for all requests, q (getData) and ANSI SQL
  • Late data can be filtered out (for example, to see only data present in the system at a particular point in time)
  • Performance is not significantly impacted by late data arrival, including for object storage tiers
  • Reference data can be queried in the Explore Window, in a data pipeline and as part of the getData() API
  • Support for non time-series reference data in Database
  • Data in reference tables can be joined to time-series tables in free-form q queries
  • Improved SQL support for queries (joins and complex queries, except for reference tables)
  • Compression support, including object storage
  • Improved resiliency and performance

[NEW] User Interface

  • Scratchpads
  • Data can be pulled into the scratchpad using either sql or getData()
  • Freeform q, SQL or Python code can be execute on that data to further filter, aggregate and join data to explore specific records
  • Python/q functions or a model can be applied to that data to gain insights

  • A model can be fitted using that data

  • Machine Learning plugins can be added to a Pipeline created from the user interface

  • More intuitive creation of data ingest pipelines
  • Observability embedded into the UI (events and logs tab)
  • Log forwarding to each providers' log aggregator is now enabled by default in the Terraform scripts.

[NEW] Reports

  • Visualization is available for ingested data using the getData() API

[NEW] Observability

  • View Database and Pipeline logs and events from the UI
  • Ability to Export Diagnostics, including the logs, events and state for further diagnosis
  • Further integration with cloud monitoring tools
  • Additional performance metrics collected

[NEW] Reliable Transport Stream

  • Subscribers have the option to override the detail truncation options for the stream logs that they have processed
  • Improved recovery and performance

Artifacts

type location
Infrastructure kxi-terraform-1.1.0.tgz
Platform insights-1.1.0.tgz
Operator kxi-operator-1.1.0.tgz
CLI kxicli-0.9.0-py3-none-any.whl
ODBC Driver kodbc 1.1.0
Java SDK java-sdk 1.1.0

Upgrade notes

Data loss of UI assembly definition

Data loss of UI assembly definitions

Assembly definitions built from the kdb Insights Enterpirse UI will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.

To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace.

kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz

To restore, the user should run the following command post-install. 

cat kxi-controller-data.tgz |  kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below 
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
  • getData does not support connecting reference data column names that do not match. If the name of a column does not match the name of the column it is a foreign key for, then the getData reference join will fail. Example: The trade table has a column sym which is a foreign key to the msym column in the table market. The join will run into an issue since the column names are different.

  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.

  • Passwords used for authenticating to third-party services are currently stored as part of the pipeline configuration. This will be corrected in the next release. It is recommended that instead of using a password directly that a Kubernetes secret is used whenever possible.

  • Currently users will be logged out but will not be notified via the UI. If users consistently encounter errors, we suggest they log out and in again.

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initialises unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if kdb Insights Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

``` 2022-05-24 20:30:48.103035 INFO 1 Started replicator to :34.142.40.180:5000, pid 41697 2022-05-24 20:31:48.127821 INFO 1 Process 41697 terminated 2022-05-24 20:31:48.128196 INFO 1 Started replicator to :34.142.40.180:5000, pid 41714 2022-05-24 20:36:12.700002 INFO 1 Process 41714 terminated 2022-05-24 20:36:12.700467 INFO 1 Started replicator to :34.142.40.180:5000, pid 41772

e set for the deployment. 2022-05-25 05:04:50.077286 INFO 0 New configuration fetched 2022-05-25 05:04:50.092259 WARN 0 Config parse error at offset 0: Expected opening bracket 2022-05-25 05:05:50.046686 ERROR 0 Config is too old 2022-05-25 05:05:50.046813 INFO 0 Waiting for the driver to sync the data ```

Backward Compatibility

  • If you're upgrading from a previous version or the CRD is already installed, helm will not update the definition automatically. See Operator Overview Helm Warning for details on how to rectify this.

License changes

From 1.1.0 kdb Insights has moved to consumption-based licensing using a new license file, kx.lic. The user should generate the license themselves using the klic self-service tool. The documentation for this is available here.

After following the klic docs, a new kx.lic should be available for your cluster.

Then you can upload it as a Kubernetes secret using the following command.

kubectl create secret generic kxi-license --from-file=license=kx.lic

Alternatively delete your existing license secret and re-run the CLI install (kxi install ..), choosing the N when prompted if you have an existing license secret. Assuming the license secret is called kxi-license, the command below will delete it.

kubectl delete secret kxi-license

To revert to the pre-1.1.0 behaviour, the following Helm values should be set for the deployment. For details on this file see here.

global:
  license:
    ..
    onDemand: true
    asFile: false

kxi-acc-svc:
  enabled: false

Single pod Storage Manager

By default, when upgrading to 1.1.0 the Storage Manager will no longer create independent containers for the various processes used to manage data persistence. These processes now all run under a single container as child processes. As part of the upgrade, it is optionally encouraged that any assembly data workflows deployed should be upgraded to align any resources with this change.

The change requires the user to remove resources configuration for eoi, eod and dbm sections of an assembly and update the sm.k8sPolicy with a resources configuration for all these processes which will run as a single container within the sm pod.

For example

  sm:
    ..
    k8sPolicy:
      resources:
        requests:
          memory: "14Gi"
          cpu: "2000m"
        limits:
          memory: "14Gi"
          cpu: "4000m"

However if no actions are taken, kdb Insights will remain backwards compatible with the existing configuration and will use the sum of all CPU and memory resources under the sm element, using this sum to populate the sm.k8sPolicy.resources section under the sm element.

1.0.0

Release for kdb Insights Enterprise.

Release Date

2022-03-14

Features

[NEW] UI allows storage and reload of data assemblies.

[NEW] Schemas can be created and deleted in the kdb Insights Enterprise user interface.

[NEW] Assemblies can be created, saved and deployed in the kdb Insights Enterprise user interface.

[NEW] Ability to Export Assembly Definitions.

[NEW] Integration with the new License Management system. Due to these changes, existing licenses will need to be re-issued. Please contact your KX representative for more information.

Artifacts

type location
Infrastructure kxi-terraform-1.0.0.tgz
Platform insights-1.0.0.tgz
CLI kxicli-0.4.0-py3-none-any.whl
Operator kxi-operator
ODBC Driver kodbc 1.0.0
Java SDK java-sdk 1.0.0

Known Issues

  • The Service Gateway may repeatably print a NullPointerException under certain conditions. If this should occur the Kubernetes pod that the service gateway process is running on should be bounced.

  • An OOM event may be triggered when repeatedly bombarded with requests that return a large result set. In the event where an OOM occurs the client will not receive a response; the request may be retried and serviced by other gateways.

  • When choosing an assembly in the Explore Window a ‘500 Internal Server Error’ can be returned if no labels are included in the labels section of the assembly definition.

  • In the prospective menu (the customizable navigation toolbar) not all options available in the menu selector are currently supported, we recommend keeping the default menu options. However, all options are included in the Entity Tree shown when the Home menu is selected.

  • When exporting an Assembly definition from the kdb Insights Enterprise UI the file requires some cleaning before it can be imported into another environment.

    • Remove id entry from tables.
    • Remove apiVersion entry from pipelines and sequencers.
    • Remove line breaks from pipeline specs.

  • When saving a pipeline in the UI on AWS or Azure you need to change the Storage Class from “standard” (which is the default for GCP) to a storage class which exists on AWS or Azure. For example, to “gp2" for AWS, or "default" for Azure. This is done in the Persistence tab when saving. Note, the Storage Class is set separately for the controller and worker. Both must be set correctly. Scroll down to set the worker Storage Class too. Otherwise, the pipeline pod and pvc deployment will get stuck in the “pending” state as no “standard” storage class can be found. Alternatively, create a “standard” storage class per the link storage class resource.

  • Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.

  • Not all stream log files written by the Reliable Transport mechanism (RT) are deleted once the stream has been replicated to the subscriber as the RT stream log archival rules are not being applied to all processes. This will cause the disk space of certain volumes associated with the kdb Insights Enterprise cluster to fill up over time. The more data that is sent through the system the faster this will occur. This may result in out of disk space issues at the producer/subscriber and in the raft cluster. We recommend that you monitor the disk usage and manually delete old stream logs regularly to prevent the disks from filling up. A tool that monitors the disk space could be used to alert you when this intervention is required (for example: Google's Cloud Watch on GCP, or Prometheus). See the section RT Logs for locations of the stream log files and how to delete old stream logs that are not longer required.

  • If a publisher is running and either the out directory on any of your 3 nodes in the RT cluster stops growing, or your Stream Processor is not receiving data, you should restart your publisher. To investigate the progress of messages in the RT cluster stream logs and look at the RT application logs see the section RT Logs.

  • If a user leaves trailing spaces after CLI fields, such as secrets, then the trailing spaces are preserved in the values.yaml file leading to errors on deployment. For example, kxi-certificate in below yaml: 
global:
caIssuer:
name: 'kxi-certificate '
secretName: 'kxi-certificate '
  • The CLI error handling is currently limited. If necessary use the underlying kubectl command to see the issue more clearly. For example, to see a yaml syntax error returned by a modified sdk_sample_assembly.yaml file, run: kubectl apply -f sdk_sample_assembly.yaml.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When using the Sample Java program, provided as part of the Java SDK, replace valFloat:float with valFloat:double in the first line of the sample.csv file before running the program.

  • The Controller expects you to call your idb, myidb, under instances if you set mounts to use different keynames than your instances an error occurs.
dap:
  instances:
    idb:
      size: 1
      mountName: myidb
      gwAssembly: insights
  • Enabling .z handler metrics accumulates memory over time and may cause containers to restart if they hit their memory resource limit.

Backward Compatibility

  • If you're upgrading from a previous version or the CRD is already installed, helm will not update the definition automatically. See Operator Overview Helm Warning for details on how to rectify this.

0.9.0

Beta 1 Release for kdb Insights Enterprise.

Release Date

2021-12-09

Features

[NEW] Run Python and q Machine Learning Models.

  • Users can add a q or Python based model to the ML registry.

  • Users can use q to retrieve either a q or a Python based model from the ML registry.

  • Users can deploy a Stream Processor pipeline that includes ML models from the ML Registry.

[NEW] Run Python functions on streaming data. Users can add Python functions into a Stream Processor pipeline to apply transformations (for example: windowing and filtering) to the streaming data.

[NEW] Support for the Kafka Schema Registry (dynamic schema updates are not yet supported).

[NEW] Users have access a single OpenAPI document. This provides documentation for the following endpoints:

  • Service Gateway

  • Client Controller

  • Information Service

  • Stream Processor

All available operations and authentication methods are documented.

[NEW] The User interface supports using ANSI SQL queries on a single database. These queries are made available via a sandbox to ensure the live database is not modified by the queries.

[NEW] Command line interface for:

  • Installation setup.

  • Enrolling and removing clients.

  • Creating and deleting assemblies.

  • Querying data.

[NEW] Login screens include KX themes

Artifacts

type location
infrastructure https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-terraform/0.1.11/kxi-terraform-0.1.11.tgz
installation https://nexus.dl.kx.com/repository/kx-insights-charts/insights-0.9.0.tgz
cli https://nexus.dl.kx.com/repository/kxi/packages/kxicli/0.1.3/kxicli-0.1.3-py3-none-any.whl

Known Issues

  • Creating a pipeline in the UI that contains an Expression node must be done through the Pipeline editor, the Import screen does not allow correct configuration of an Expression node.

  • If one of the nodes in an RT sequencer is killed or restarts, data may no longer flow through that RT. If the issue occurs on north and south sequencer, it may be sufficient to restart the assembly. Otherwise, the cluster needs to be restarted.

  • RT nodes can under certain conditions consume excessive CPU causing the cluster CPU go up to >95%. If the issue occurs open a shell to the affected node(s) and kill all the pull_server processes. They will be automatically restarted and the CPU usage will drop.

  • RT occasionally reports an inotify error. This is a resource exhaustion issue. If the issue occurs on north and south sequencer, it may be sufficient to restart the assembly. Otherwise, the cluster needs to be restarted.

  • Both SQL and q queries work with assemblies. However, when running a query against a html5evalcongroup process in the UI, SQL is not supported and q must be used instead.

  • Programmatic OpenAPI kdb Insights client generation currently fails due to an issue with the Stream Processor.

Backward Compatibility

  • If you're upgrading from a previous version or the CRD is already installed, helm will not update the definition automatically. See Operator Overview Helm Warning for details on how to rectify this.

0.8.0

Initial Alpha Release for kdb Insights Enterprise.

Release Date

2021-10-27

Features

[NEW] Secure login & authentication using Keycloak, enabling enrollment and removal of client publishers.

[NEW] Role Based Access Control with roles for pipeline viewing and editing, access to the user interface and the ability to query data.

[NEW] Licensing is via a base install key.

[NEW] Data ingestion via:

  • Publishing from the RT command line interface.

  • Reading from Kafka.

  • Publishing using direct qIPC.

  • Batch ingestion from S3 buckets.

[NEW] The User interface supports creation of stream processor pipelines; including data ingestion and transformation, and presentation in grids/charts. For this release, pipelines can be published, but drafts are only saved to the user’s browser - not to kdb Insights Enterprise.

[NEW] Data transformations can use q

[NEW] Data transformations can use a limited Python API (built on embedPy)

[NEW] Data can be streamed out via Kafka.

[NEW] Data can be visualized using the kdb Insights user interface

[NEW] Data stored in the kdb databases can be accessed via:

  • The user interface using q

  • A REST API

[NEW] Application Metrics can be generated which can be monitored via Prometheus and Grafana.

Artifacts

type location
infrastructure https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-terraform/0.1.7/kxi-terraform-0.1.7.tgz
installation https://nexus.dl.kx.com/repository/kx-insights-charts/insights-0.8.0.tgz

Known Issues

  • On AWS - deployment of infrastructure needs to be targeted at a region that has 3 availability zones - see https://aws.amazon.com/about-aws/global-infrastructure/regions_az/

  • On AWS(us-east-2) - Due to an issue specific to us-east-2 terraform tries to fetch ec2 information by hitting ec2.us-east-2.amazonaws.com normally this url should return a single public ip address but within an ec2 instance on us-east-2 region it returns 3 private ips which are not accessible via https. For this reason the application should not be deployed to us-east-2.

0.10.0

Beta 2 Release for kdb Insights Enterprise.

Release Date

2022-02-04

Features

[NEW] Support for the Kafka Schema Registry has been extended to include dynamic schema updates.

[NEW] Defining ML models can be done using Python.

  • Users can use Python to retrieve either a q or a Python based model from the ML registry.

  • Users can use Python to save either a q or a Python based model from the ML registry.

[NEW] The ANSI SQL support has been extended to include distributed databases. Currently joins are not supported and tables with the same names in different databases MUST have the same columns and properties. As in the previous release these queries are made available via a sandbox to ensure the live database is not modified by the queries.

[NEW] REST API getData function has been enhanced to include:

  • Basic aggregation functions on individual columns (for example: min, max, avg, stdev)
  • Racking and Filling
  • Filtering
  • Time range options, including temporaility

[NEW] Schemas, Streams and Databases can be created and deleted in the kdb Insights Enterprise user interface.

[NEW] KX ODBC driver for data ingestion. Enabling data ingestion by applications that support ODBC.

[NEW] KX Java SDK for data ingestion.

Artifacts

type location
infrastructure kxi-terraform-0.1.12.tgz
installation insights-0.10.0.tgz
cli kxicli-0.2.0-py3-none-any.whl
operator kxi-operator-0.2.0-rc.19.tgz
ODBC driver RPM package kodbc-0.1.52-rhel.x86_64.rpm
ODBC driver Debian kodbc-0.1.52.deb
ODBC driver Windows executable kodbc-0.1.52.exe
ODBC driver Windows msi kodbc-0.1.52.msi
ODBC driver sample program kodbc-0.1.52.zip
Java SDK rt_client-1.0.0rc6.jar
Java SDK sample program rtdemo-1.0.0rc6-all.jar
Java SDK sample csv file sample.csv

Known Issues

  • The sample assembly yaml files provided specify the same subTopic: "data". To start multiple assemblies in parallel, change the subTopic to a unique value, such as subTopic: "odbc-assembly". The corresponding change will also need to be made when enrolling the client. For example: curl https://${INSIGHTS_HOSTNAME}/clientcontroller/enrol -H "Authorization: Bearer " -d '{ "name": "my-client", "topics": { "insert": "odbc-assembly", "query": "requests" }}'

  • The UI may produce pop-up error messages in various scenarios.

  • The UI does not have Reports available.

  • kdb Insights CLI error handling is currently basic. If the CLI throws a stack trace, use the underlying kubectl command to see the issue more clearly. For example, to see a yaml syntax error returned by a modified basic_assembly.yaml file, run: kubectl apply -f basic_assembly.yaml .

  • KX ODBC driver and Java SDK do not work when using kdb Insights Enterprise installed on an AWS cluster.

Backward Compatibility

  • If you're upgrading from a previous version or the CRD is already installed, helm will not update the definition automatically. See Operator Overview Helm Warning for details on how to rectify this.