Skip to content

kdb Insights Enterprise - Previous Release Notes

This page outlines release notes from past releases. For the most recent release, refer to the latest release page.

1.9.4

Release Date

2024-06-14

Fixes

Expand to see the full detail of resolved issues here

Security

  • Multiple CVEs have been resolved as part of this patch.

Cloud Readers

  • When administering a pipeline, a cloud reader node (s3, ms, and gs) could prompt you that there were unsaved changes even when no changes have been made.

  • The Azure reader file watcher was returning an error that one or more files did not exist despite the file(s) being present.

Reliable Transport

  • RT's replicator was emitting an Invalid publishing directory, was it deleted by mistake? Please enrol a different client and restart the SDK with the new client error after recovering from a power outage of a q publisher server. This has been resolved to handle the recovery cleanly after such an outage.

UI

  • On the Pipeline page, an intermittent issue could cause performance degradation or unresponsiveness in the UI for extended periods.

  • When configuring a pipeline in the UI, setting additional workerThreads was not correctly applying the specified configuration. However, setting the environment variable KXI_DEFAULT_SP_WORKER_THREADS did work. This has been resolved so either method can be used.

  • The default setting for retries for database reader/writers in the UI did not match the backend defaults. This has been corrected to be consistent, with the default setting of 60 seconds for retries, and a retry wait time of 3 seconds.

  • When performing a rollback, the CLI was deploying the rollback task with a hard-coded timeout. This has been changed to a configurable value for use when rollbacks are expected to take longer.

  • When deploying a database, the Deploy popup was displaying totals that did not match the selected CPU, RAM and disk settings. The same issue was present in the Resource Summary box on the main page. These calculation errors have been resolved to display correct totals.

Database

  • For a new system with an empty database, queries would not work after the first end-of-day data writedown if no data had been ingested yet.

Third-party Dependencies

kdb Insights Enterprise 1.9.4 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.9.1 release of the standalone infrastructure installation scripts.

1.9.3

Release Date

2024-05-21

Fixes

Expand to see the full detail of resolved issues here

Reliable Transport

  • RT's replicator was emitting an Invalid publishing directory, was it deleted by mistake? Please enrol a different client and restart the SDK with the new client error after recovering from a power outage of a q publisher server. An intermittent issue with truncated log files when the publisher restarts has been resolved.

Third-party Dependencies

kdb Insights Enterprise 1.9.3 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.9.1 release of the standalone infrastructure installation scripts.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type Nexus location Downloads Portal location
Enterprise insights-1.9.3.tgz insights-1.9.3.tgz
Operator kxi-operator-1.9.1.tgz kxi-operator-1.9.1.tgz
CLI kxicli-1.9.1-py3-none-any.whl kxicli-1.9.1-py3-none-any.whl
RT C interface kxi-c-sdk 1.9.0 kxi-c-sdk 1.9.0
RT Python interface kxi-rtpy-1.9.0 kxi-rtpy-1.9.0
RT Java interface kxi-java-sdk 1.9.0 kxi-java-sdk 1.9.0
RT Bridge rt-bridge 1.7.0 rt-bridge 1.7.0
Infrastructure kxi-terraform-1.9.1.tgz kxi-terraform-1.9.1.tgz
kxi-management-service kxi-management-service-0.2.1.tgz kxi-management-service-0.2.1.tgz
insights-on-k8s insights-on-k8s-0.0.16.tgz insights-on-k8s-0.0.16.tgz

1.9.2

Release Date

2024-05-07

Fixes

Expand to see the full detail of resolved issues here

Reliable Transport

  • The RT client was failing to recognize a new log file, putting downstream subscribers into a hung state. This has been resolved.

Third-party Dependencies

kdb Insights Enterprise 1.9.2 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.9.1 release of the standalone infrastructure installation scripts.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type Nexus location Downloads Portal location
Enterprise insights-1.9.2.tgz insights-1.9.2.tgz
Operator kxi-operator-1.9.1.tgz kxi-operator-1.9.1.tgz
CLI kxicli-1.9.0-py3-none-any.whl kxicli-1.9.0-py3-none-any.whl
RT C interface kxi-c-sdk 1.9.0 kxi-c-sdk 1.9.0
RT Python interface kxi-rtpy-1.9.0 kxi-rtpy-1.9.0
RT Java interface kxi-java-sdk 1.9.0 kxi-java-sdk 1.9.0
RT Bridge rt-bridge 1.7.0 rt-bridge 1.7.0
Infrastructure kxi-terraform-1.9.1.tgz kxi-terraform-1.9.1.tgz

1.9.1

Release Date

2024-04-30

Improvements

  • The fromParquet Reader is now supported for AWS-based deployments. For more information, refer to Readers.

  • Streams can now be removed from packages using the rm command from the CLI. Run kxi package rm --help in the CLI for details.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Client requests using compressed IPC messages are now properly serviced. Previously, if the first request from a client was using compressed IPC, that and any future requests would fail.

Terraform Scripts

  • Previously, in the kdb Insights Enterprise Terraform sample client scripts, the package-manager-migration command failed when used in Azure deployments using the sharefiles storage class. This has been resolved. New sample scripts are available as part of this release. See the Artifacts section below.

Stream Processor

  • Pipelines now throw an error when writing to a non-existent table. Previously, no errors were present in the pipeline or database logging. This was most commonly experienced when renaming a table in the database without updating the pipeline as well.

Reliable Transport

  • Reliable Transport pods now use Kubernetes liveness probes to automatically bounce pods if unresponsive. This resolves a known issue in 1.9.0 in which RT pods were required to be manually restarted if an upgrade erroneously left them in a non-ready state.

Packaging

  • The UI now displays the running version of a package as opposed to the latest package that may be available.

  • subTopic has been added as a mandatory field when adding external streams to a package. Previously, this was not present and caused a package to be undeployable.

  • In the UI, it is no longer possible to create pipelines with the duplicate names.

  • The CLI now errors when trying to create and save a package that contains duplicate UUIDs, or when trying to save entities within a package that have duplicate UUIDs.

UI

  • Resource allocations are no longer mandatory input fields on the Resources tab. It is now possible to leave them empty and submit your resource configuration. For more information, refer to Adjust resources.

Third-party Dependencies

kdb Insights Enterprise 1.9.1 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.9.1 release of the standalone infrastructure installation scripts.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type Nexus location Downloads Portal location
Enterprise insights-1.9.1.tgz insights-1.9.1.tgz
Operator kxi-operator-1.9.1.tgz kxi-operator-1.9.1.tgz
CLI kxicli-1.9.0-py3-none-any.whl kxicli-1.9.0-py3-none-any.whl
RT C interface kxi-c-sdk 1.9.0 kxi-c-sdk 1.9.0
RT Python interface kxi-rtpy-1.9.0 kxi-rtpy-1.9.0
RT Java interface kxi-java-sdk 1.9.0 kxi-java-sdk 1.9.0
RT Bridge rt-bridge 1.7.0 rt-bridge 1.7.0
Infrastructure kxi-terraform-1.9.1.tgz kxi-terraform-1.9.1.tgz

Upgrade Notes

Upgrading to 1.9.x

  • Before you upgrade to 1.9.x, if you have any running databases, it is advised to manually stop them before upgrading, otherwise you may become unable to edit them in the UI. If this happens, you can recover the editable state by considering the following options:

    • If possible, back up the running assembly yaml file with kubectl get asm
    • Tear down the running assembly manually with kubectl delete asm
    • Optional: use the kxi package deploy command in the CLI, specifying the name and version of the package. The package should appear under kxi package remote-list. Refer to Package deployment for more information.

    If the package doesn't reappear as editable after tearing down, you need to use the KXI CLI to perform the following operations:

Known Issues

Expand to see the list of known issues here

Custom Tier Names

  • Existing assemblies that have non-default tier names (i.e. names other than "rdb","idb","hdb") that are converted to packages will need to be updated to use those defaults. Presently, the UI expects the default names and is unable to differentiate between hdb and additional tiers.

1.9.0

Release Date

2024-03-26

New Features

  • Developer Experience Improvements for Packaging

    kdb Insights Enterprise packaging now allows clients to manage database and pipeline configuration as well as custom code within a single package. This greatly simplifies developer and operational workflows, as users can maintain and administer databases and pipelines within a single package and with specific configuration for each pipeline. Packaging has also been integrated with the UI so that any databases and pipeline created using the CLI using the kxi package command can be viewed and maintained by the UI and vice versa. Refer to Packaging for more details.

  • SQL Enhancements

    SQL in kdb Insights Enterprise now includes support for ORDER BY and LIMIT clauses, in addition to other functions. Changes to how SQL statements are parsed also significantly improves SQL query performance in this release. Query performance, on average, is up to 3x faster than before. Refer to SQL 2 for more information on the new SQL implementation and its current Known Limitations.

  • Streaming to Views

    In the kdb Insights Enterprise UI, Views can now subscribe to real-time data from a pipeline, rather than relying on intermittent polling. This gives you the ability to visualize and analyze your data immediately. Refer to the streaming guided tour for details on configuring pipelines and views for this feature.

  • Snapshots & Majority Archival for Reliable Transport

    Reliable Transport now supports taking snapshots allowing for majority archival of logs files. Previously, all nodes needed to be healthy before RT logs could be archived.

  • Configurable Connections for Reliable Transport

    Reliable Transport now supports subscribers and publishers setting customized parameters when they register with their respective sequencer cluster. This gives you greater configurability in how you deploy and orchestrate components of RT. See rt.qpk for more details.

Improvements

  • System Resources Configuration

    Defining system resources has been streamlined, enabling you to set all aspects of resource allocation from a single configuration page in the kdb Insights Enterprise UI. See Adjust Resources for more information on this new tab.

  • Batch Delete

    Batch delete now allows you to delete from all tiers of your HDB. Previously, this was only possible for the tier with the most recent HDB data. See Database Delete for details on how to perform a delete.

  • Python Cloud Readers

    Python cloud readers now support polling for new matches to a glob pattern with the watch parameter. See Readers for details.

  • Upgrade to Insights Core 4.1.1

    kdb Insights Enterprise images have been upgraded to run Insights Core 4.1.1, which runs kdb+ 4.1 2024.03.12.

  • Upgrade to RockyLinux 9

    kdb Insights Enterprise images have been upgraded to run on Rocky Linux version 9 as their base operating system. This addresses security vulnerabilities present in version 8.

  • Stream Processor Troubleshooting Guide

    There is now a troubleshooting guide{:target="blank"} available to help users better understand how to debug their pipelines running in _kdb Insights Enterprise.

Fixes

Expand to see the full detail of resolved issues here

Database

  • If EOD failed while in the middle of an HDB transaction, the aborted transaction was not properly cleaned up after the failure and could result in data inconsistency after reconnection.

  • The EOD process was not performing a garbage collection after a batch ingest.

  • Data was unqueryable after a batch ingest if no additional streaming data was loaded after the initial batch ingest.

Reliable Transport

  • In deployments with a one-node RT configuration, RT has been updated to roll its log file after a restart. This prevents data flow from being paused. The RT log file will roll from log.x.y to log.x.y+1.

UI

  • The Subscriber node was limited to only specifying one keyed column. This has been resolved to allow one or more keyed columns, ensuring that streaming supports a record being streamed per unique combination of all specified keyed columns and to allow filtering on any one of the keyed columns.

  • When configuring a Schema node, the regular expression pattern for table validation has been updated to allow single character column names.

  • For Function nodes, users can now customize their function parameters to accept the metadata and operator, in addition to data, in the Pipeline UI. See here for details.

  • The Error Mode in the Pipeline Settings section of the Pipeline editor has been removed as it is no longer required, due to a stack trace now being logged when user code errors.

  • The logging level within the Runtime Pipeline Settings section of the Pipeline editor was not correctly displaying or setting the log level of the SP worker. This has been resolved.

  • There was no way to specify \t as the delimiter in a CSV decoder in the UI. This has been corrected in the CSV Decoder node to allow for any delimiter to be used.

  • After a pipeline has been deployed, when the status is either “Running” or “Finished”, a user is able to delete the pipeline. When the pipeline is deleted, the Overview page was still displaying it under “Running Pipelines”. This has been resolved to not display deleted pipelines.

  • Disk sizes for IDB and HDB tiers can now be set to any value greater than 0.

  • Users without the permission to create a pipeline should not be able to use the Import Data button on Database Page. This has been resolved.

  • Deleting running databases is now allowed.

Views

  • Notifications in Views were not reporting correctly. This has been resolved.

  • Under certain circumstances when you had a particular data source selected in the Data Source window and clicked New the existing values remained visible, however they were all ignored. This has been resolved.

  • When using the getData API with agg set to sym;distinct;sym, the UI was ignoring this request and not sending any aggregation. This has been resolved.

  • Switching between tabs caused the user-selected date to switch back to the default setting. This caused the View to constantly reload. This has been resolved.

Stream Processor

  • When using Python, 'sp.decode.csv' did not support reading columns as strings. This has been updated to enhance flexibility when reading table columns. It now accepts an empty string "" in lieu of a schema to read in all table columns as strings.

  • Sidecar containers no longer log at a TRACE level by default and now adhere to the log level that is set by the user.

  • The SP controller now adheres to the log level set by the user.

  • The CSV decoder did not support decoding csvs which had line-returns (i.e. \n’s) embedded in strings. This functionality can now be turned on with a new optional parameter called Strings Contain Newlines in the UI and newlines in the q and Python APIs.

  • When running a large import from a static source (file, Azure, AWS, GCP), there was no indication of progress in the logs. The SP logs now displays the file offset periodically while processing to indicate on-going progress of the import.

  • The SP’s status APIs will now return TEARING_DOWN rather than NOTEXIST when a pipeline has PVCs that are still in a terminating state.

  • Under certain circumstances when you added an additional stream to your database it could become orphaned after you renamed the database. This has been resolved.

  • Under certain circumstances when creating a pipeline using the Import feature the Parse Strings column was missing from the Apply Schema page, causing the pipeline to fail when deployed. This has been resolved.

  • The running pipelines section displays a count of warnings and errors that have occurred in the last 10 minutes. Not all of these warnings were being displayed in the Diagnostics page. This has been resolved.

CLI

  • When trying to enroll a client with a topic, the client was enrolled but the specified topic was not being set. This has been resolved.

  • When a user tried to authenticate but did not have a local browser installed on their machine, the redirect URL was localhost instead of the value specified. This has been resolved such that users with a local browser installed can still successfully authenticate.

  • Installation with a kc.lic license using the --license-filepath command-line parameter was generating an incorrect values.yaml file. This has been resolved so the license values correspond to the license name.

  • Specifying a different hostname as part of a kxi command is ignored after authentication, as you can only be authenticated to one kdb Insights Enterprise host at a time. For example, if you authenticate to HOSTNAME1 by calling kxi --hostname HOSTNAME1 auth login you are free to make calls against this host. If you attempt to specify a different hostname in subsequent calls kxi --hostname HOSTNAME2, the --hostname parameter will be ignored and the command executed against HOSTNAME1. You must re-authenticate against HOSTNAME2 before running commands against it.

  • The function .kxi.udfs.list.all[] was failing with a length error. This has been resolved.

  • Global environment variables set within your values.yaml will now be set by all database and assembly deployments.

    global:
    env:
        env: var
    

Query Environments

  • Query environments with entitlements enabled required the creation of separate entitlement for the -qe assembly in order for queries to work. This has been resolved.

Upgrades

  • Fixed an issue where it was possible for pods to be unscheduled after an upgrade or rollback due to a volume node affinity conflict. Please note that this is still possible when rolling back from 1.9 to earlier versions. Please check the release notes of the target rollback version to address.

Security

  • Various CVEs have been remediated as part of this release.

Third-party Dependencies

kdb Insights Enterprise 1.9.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.9.0 release of the standalone infrastructure installation scripts.

Deprecation

  • The CLI kxi assembly command is being deprecated in 1.9.0 in favor of kxi package and will be removed in a future version. See kdb Insights CLI Reference for more information on the kxi assembly command.

  • prtnEndCB and reloadCB callback functions are being deprecated in 1.9.0 and may be removed from future releases. They have been replaced by lifecycle.partition.end and lifecycle.reload.post respectively. Their usage can be seen below.

.custom.da.prtnEndCB:{[event;data]
  log.info("lifecycle.partition.end invoked.")
 }
.custom.da.reloadCB:{[event;data]
  log.info("lifecycle.reload.post invoked.")
 }

.ev.add[`lifecycle.partition.end ;`.custom.da.prtnEndCB];
.ev.add[`lifecycle.reload.post ;`.custom.da.reloadCB];
  • RT maxLogSize is no longer used in configuration and will be ignored from older configurations. RT will now utilize all disk available to it.

Artifacts

KX Nexus

The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.

type Nexus location Downloads Portal location
Enterprise insights-1.9.0.tgz insights-1.9.0.tgz
Operator kxi-operator-1.9.0.tgz kxi-operator-1.9.0.tgz
CLI kxicli-1.9.0-py3-none-any.whl kxicli-1.9.0-py3-none-any.whl
RT C interface kxi-c-sdk 1.9.0 kxi-c-sdk 1.9.0
RT Python interface kxi-rtpy-1.9.0 kxi-rtpy-1.9.0
RT Java interface kxi-java-sdk 1.9.0 kxi-java-sdk 1.9.0
RT Bridge rt-bridge 1.7.0 rt-bridge 1.7.0
Infrastructure kxi-terraform-1.9.0.tgz kxi-terraform-1.9.0.tgz

Upgrade Notes

  • Keycloak and Postgres Values Changes

    The values file settings for the Keycloak and Postgres components have changed for 1.9. The panels below describe the modifications you need to make to this file for all standard or shared Keycloak deployments before upgrading to 1.9.

    Merging install values

    The values below are a partial extract of your deployment values. These should be merged with your existing values file.

    For more information, refer to Applying configuration changes.

    A standard deployment deploys Keycloak as part of the application. If you're using this configuration, you should ensure your values file includes the details shown below.

    The names kxi-keycloak and kxi-postgresql are the default values, update these as appropriate.

    global:
      keycloak:
        auth:
          existingSecret: kxi-keycloak
      postgresql:
        auth:
          existingSecret: kxi-postgresql
      ..
    
    keycloak:
      auth:
        existingSecret: kxi-keycloak
      postgresql:
        auth:
          existingSecret: kxi-postgresql
    ..
    

    Keycloak values

    Please note there are two sets of keycloak and postgres objects; global.keycloak, global.postgresql, keycloak, keycloak.postgresql.

    Both sets of these are required as shown above.

    If deploying using the shared Keycloak mode, configure the values file as described below.

    The KEYCLOAK_RELEASE_NAME and KEYCLOAK_NAMESPACE variables should be updated to match those of your Keycloak deployment. The shared Keycloak docs explain each of these values.

    global:
      keycloak:
        authURL: http://${KEYCLOAK_RELEASE_NAME}.${KEYCLOAK_NAMESPACE}.svc.cluster.local/auth/
        auth:
          existingSecret: kxi-keycloak
      postgresql:
        auth:
          existingSecret: kxi-postgresql
      ..
    
  • If a user has an existing kxi-license secret created from a kc.lic or k4.lic, kxi install setup will re-use this secret, but will not set the appropriate values in the resulting values.yaml file. See available license types Setup - kdb products.

  • The default image repository used for new 1.9 installations will be the KX Downloads Portal. The credentials to access this are stored in a docker config json secret named kxi-image-pull-secret. If you are upgrading from an earlier version that used the KX Nexus image repository with an exiting secret kxi-nexus-pull-secret, this existing repository and secret will be used.

  • After executing an upgrade, if an RT pod does not appear as ‘Ready’, the pod should be deleted/killed so that it is restarted.

  • During an upgrade, the RT pods attempt to delete a Kubernetes configmap. Since there were 3 RT pods, only one of the pods will be successful in deleting the config map. This means that the other pods will receive ERRORs back indicating that their delete attempt failed. These Failed to delete snapshot: tmp errors are benign and will be set to WARN messages in future.

  • Query Labels Style

    KXI_ALLOW_OLD_LABEL_STYLE now defaults to "false". It must be explicitly set to "true" in the RC to allow the "old label style". For example, in a system with labels region, and class:

    • Labels in queries (such as .kxi.getData) must be defined in the labels parameter.

      • old style: --data '{"table": "trade", "region": "usa", "class": "equities"}'
      • new style (new default): --data {"table": "trade", "labels": {"region": "usa", "class": "equities"} }'
    • Labels in SQL queries must be prefixed with label_.

      • old style: SELECT * FROM trade WHERE region='usa', class='equities'
      • new style (new default): SELECT * FROM trade WHERE label_region='usa', label_class='equities'

Rollback

  • If performing a rollback from 1.9.0 to any earlier version, assemblies must be running prior to initiating the rollback to ensure no data is lost. See Upgrading for how to perform upgrades/rollbacks and the available commands to use. Please see the following specific instructions per version:

    Rollback from 1.9 to a version less than 1.7 is not possible.

    For rollback from 1.9 to a prior version:

    1. Confirm all assemblies are running before initiating rollback.
    2. Execute the rollback operation. Note that the kxi-rt-rollback and kxi-sp-rollback charts are run automatically as part of the rollback operation as of 1.9.0.

    For rollback from 1.8, the following instructions should be followed:

    1. Teardown your assemblies.
    2. Run the kxi-sp-rollback chart. If the target version for rollback is less than 1.7, also run the kxi-db-rollback chart.
    3. Rollback kdb Insights Enterprise per the rollback instructions referenced above.
    4. Redeploy assemblies.

Known Issues

Expand to see the list of known issues here

Upgrades & Rollbacks

  • Users are prompted to tear down assemblies upon rollback. The rollback should exit if the user responds ‘n' to this prompt. Instead, the rollback continues.

  • Occasionally, rollbacks may fail with an error when running kxi install rollback. This is due to timeouts with the Helm rollback, resulting in errors such as the following:

        timed out waiting for the condition
        context deadline exceeded
    

    To resolve this issue, re-run the rollback.

Pipelines

  • It is currently possible to rename a pipeline in the UI to a pipeline name that is already in use. Until this is resolved, ensure your pipeline names are unique.

  • When deploying a pipeline that writes to a table that does not exist, no error will be thrown.

Packaging

  • kxi.package.load_file does not work properly for Python files. For a Python module, use import instead.

Query Tab

  • There is presently no way to select a subset of the available columns during the getdata phase in the Query tab of the UI. Conversely, the getData API allows this in the agg parameter where you can pass in the list of columns you want with no aggregation function.

UI

  • Changing the system log level in the UI does not get actioned.

  • On Mozilla Firefox, when clicking Deploy in a Database screen, the scroll wheels do not render properly in the pop-up.

Views

  • When displaying streaming data in Views and using the Map component, the map may take some time before the streaming data is displayed. A page reload may be required.

  • When working with a View then temporarily navigating away to a different tab, any changes made will be saved despite the user not requesting this. If multiple users are editing the same view, any time one switches away from the tab (even if they haven’t made any changes), the current version of their view will be saved, which may not include changes made other users. To resolve this, use the Undo action to return the last desired saved state.

  • A popup may be shown indicating there may be unsaved changes when no changes have been made.

Scratchpads

  • Certain Python expressions cause internal errors in Python scratchpads:

    lambda x: x * 2
    memoryview(bytearray("myTest", encoding='utf-8'))
    {1,2,3}
    

Reliable Transport

  • RT clients output certain logs to a folder determined by theKXI_C_SDK_APP_LOG_PATH environment variable ($KXI_C_SDK_APP_LOG_PATH/kxi_c_sdk_logs/rt_helper.VMC7AW/rt_helper.log). This variable uses a default location of /tmp. When there are recurring errors, these logs can fill the /tmp directory which is only 5MB in size. To overcome this issue, you can change this environment variable. See Environment variables for steps on how to change this.

Deprecation

  • The CLI kxi assembly command is being deprecated in 1.9.0 in favor of kxi package and may be removed from future releases. See kdb Insights CLI Reference for more information on the kxi assembly command.

  • prtnEndCB and reloadCB callback functions are being deprecated in 1.9.0 and may be removed from future releases.

  • RT maxLogSize is no longer used in configuration and will be ignored from older configurations. RT will now utilize all disk available to it.

1.8.2

Release Date

2024-01-23

Fixes

Expand to see the full detail of resolved issues here

Database

  • An issue has been resolved in relation to Batch Ingest. Previously, some data in the Historical Database (HDB) was persisted to disk but it was unqueryable. This has been fixed, and all data ingested into the HDB is now queryable.

Third-party Dependencies

kdb Insights Enterprise 1.8.2 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.8.2 release of the standalone infrastructure installation scripts.

Artifacts

Type Location
Enterprise insights-1.8.2.tgz
Operator kxi-operator-1.8.1.tgz
CLI kxicli-1.8.0-py3-none-any.whl
C interface kxi-c-sdk 1.8.0
RT Python interface kxi-rtpy-1.8.0
Java interface kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.7.0
Infrastructure kxi-terraform-1.8.0.tgz

1.8.1

Release Date

2024-01-16

Improvements

  • Encryption of data in transit with OpenShift

    The encryption of data in transit is now supported when deploying kdb Insights Enterprise using OpenShift.

  • Stream Processor CSV Decoder Improvements

    The Python CSV decoder now supports the first header option. This should be used when only the first batch of your dataset contains a header in the CSV file. See Decoders for information on the CSV decoder configuration.

Fixes

Expand to see the full detail of resolved issues here

Packaging

  • Resolved an issue where .kxignore files were incorrectly read as bytes instead of strings when compressed on Ubuntu for Windows.

Stream Processor

  • Stream Processor Beta features configuration has been fixed. When deploying the SP Helm chart, setting betaFeatures: true wasn't correctly enabling Beta features.

  • A fix has been introduced in the UI to ensure changes to the position of an SP node in a pipeline are now correctly saved.

  • When creating a pipeline in the UI, the Stream Processor S3 writer now allows for proper writing to an object storage bucket. Previously, this could error with Some nodes have errors, fix them before trying to deploy this pipeline.

  • Previously, the CSV decoder could drop messages when paired with a file reader (Amazon S3, Google Storage, Azure Storage) where chunking was enabled and the CSV decoder header was set to always. This issue has been resolved. See Decoders for information on the CSV decoder configuration.

  • SP Cloud Readers no longer error when there are no files matching the expected pattern. Previously, a cloud reader using glob pattern matching would prevent the pipeline from starting if no files matched. In particular, this would occur when trying to use the file watcher feature and no files matched as of the pipeline starting.

  • The delimiter select field in the Stream Processor CSV node has been updated to allow any delimiter to be used. Previously, there was a restriction on the number of characters that could be submitted, preventing entry of \t or any other delimiter.

  • Resolved the problem of all window nodes triggering their jobs at the same time, despite their configuration settings. This would occur in setups with pipelines with multiple window nodes, or separate pipelines that read and write to the same RT stream. See Windows nodes for more details.

Security

  • Various CVEs were remediated as part of this release.

Third-party Dependencies

kdb Insights Enterprise 1.8.1 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.8.1 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.8.1.tgz
Operator kxi-operator-1.8.1.tgz
CLI kxicli-1.8.0-py3-none-any.whl
C interface kxi-c-sdk 1.8.0
RT Python interface kxi-rtpy-1.8.0
Java interface kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.7.0
Infrastructure kxi-terraform-1.8.0.tgz

1.8.0

Release Date

2023-12-11

New Features

  • Data Encryption

    Data in transit within kdb Insights Enterprise can now be encrypted, providing enhanced security. This feature meets compliance and regulatory requirements and protects against cyber-security attacks. For further details on how to enable encryption via the CLI, see Encryption of data in transit.

  • OpenShift Container Platform (OCP) Support

    kdb Insights Enterprise can now be deployed on premise using the OpenShift Container Platform (OCP). This expands on the deployment topology options of kdb Insights Enterprise, and enables customers to take advantage of its capabilities without having to run in a public cloud. For further details, see OpenShift infrastructure prerequisites.

    Note

    See here for details on an issue with Openshift upgrades.

  • Azure and GCP Parquet Reader Support

    The kdb Insights Parquet Reader functionality has been extended to support Azure and GCP object storage. Users can now read, decode, and watch Parquet files stored on Azure and GCP, as well as AWS. These enhancements are available in the UI, the q API and the Python API.

  • HTTP Publishing

    kdb Insights Enterprise has introduced a REST interface to publish data over HTTP. This expands the interoperability of kdb Insights Enterprise for ingesting data. Note that file size is limited to 10MB per publish. See Upload via HTTP for more details.

  • Python Interface to Reliable Transport

    A new Python interface allows you to publish data to Reliable Transport from your own Python application in Docker, or on Linux or Windows. See here for details.

  • New Database Preview API

    You can now query a sample of your data from the database to understand your data/schema, construct more complex queries, or build visuals on the full data set. See here for details.

Improvements

  • Autoscaling Service Gateways

    The Service Gateway and Aggregator can now autoscale if CPU exceeds your configured threshold. The default is to trigger scaling when CPU exceeds 80% for 5 seconds, with a default stabilization period of 300 seconds. All of this is configurable in your values file. See here for details.

  • CLI Enhancements

    The kdb Insights Enterprise CLI contains a number of security and user experience improvements added:

    • Browser-based authentication for users, and the ability to authenticate even without a browser
    • Ability to authenticate once and still straddle administrator and developer views
    • Token caching & refreshing to support the authentication methods
    • Use of profiles to allow for concurrent sessions in different environments

    See CLI Authentication for details.

  • UI Enhancements

    • When editing a pipeline node, you are now reminded to apply your current changes before moving away to perform other operations. This avoids the potential loss of changes when clicking on to another node.

    • Ability to view the rolled-up status of a pipeline group when there are multiple pipelines/replicas.

    • Undo and Redo buttons have been added to the Pipeline editor.

    • Pipeline settings have been moved into a new ‘Settings’ tab on the Pipeline page. See here for details.

    • For tables marked in a foreign key relationship, further validation has been added upon startup to check that these exist and fail fast if any are missing.

    • The autocomplete widget in code editors now only appears when manually triggered with Ctrl + Space.

  • Stream Processor

    • Cloud file readers in the Stream Processor now support watching for new files. You can now provide a pattern expression to search for file names. When watching is enabled, the file reader will poll for new files that match the expression. For full details on file watching, see the Amazon S3 reader, the Azure Blob storage reader and the Google Cloud storage reader.

    • You can now deploy replicas of pipelines. When deploying, select the “Advanced” option to configure a “Replicas” input field. The environment will deploy the configured number of replicated pipelines identical in configuration with a different ID. See here for details.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Improved startup performance of HDB DAPs when loading late data from IDB tier mount.

  • Storage Manager resiliency improvements have been made to improve how SM cleans up aborted transactions after a failure event.

  • Improved validation for schemas in the UI. Rules now validate against invalid configuration combinations with attributes, sorts and reserved words.

  • Improved failure recovery for object storage in Storage Manager. This resolves a case where EOI would not complete and would report SM Flushing filesystem with [directory] errors.

Reliable Transport

  • Logging for LEADER/FOLLOWER updates have been changed to DEBUG level to reduce log noise.

Stream Processor

  • Previously, it was possible that FINISHED SP pipelines using multiple partitioned workers could spawn extra zombie workers. This has been resolved.

  • Pipeline quick tests no longer fail if they contained a kdb Insights Stream or kdb Insights Database reader or writer.

  • Previously it was not possible to maintain state of any object that contained a kdb guid type. This is now fixed. However, if a rollback to an earlier version is required after an upgrade, a rollback step should be run so that on-disk checkpoints can be converted to a format compatible with earlier versions.

  • Fixed an issue where the csv decoder, with option header=first, and paired with an unbounded streaming reader, was incorrectly dropping the first record in every stream batch.

  • Fixed a determinism gap when reading objects from cloud storage using a glob pattern. In a recovery scenario, it was possible to receive duplicate data.

  • Fixed a bug where unpopulated column of type 0h (i.e. list) was not getting filled in with blank entries correctly.

  • If a pipeline is deployed as part of an assembly, .qsp.write.toDatabase now defaults to writing to that assembly. This means the assembly name can be omitted when calling .qsp.write.toDatabase, for example, .qsp.write.toDatabase[trade; .qsp.use`directWrite!(::;1b)].

  • Fixed a bug where in a single path pipeline, the existence of more than one emitter resulted in that same number of combine nodes being added.

Upgrades

  • Resolved a previously-published known issue where upgrades could occasionally fail with an error when running kxi install upgrade. This occurred due to a race condition in the 1.7.0 version of the CLI when trying to replace the CRDs.

File Watchers

  • When using the file/cloud readers in a non-file watching scenario, files that were not getting chunked did not have their state checkpointed. This meant that during a recovery scenario, duplicate data would be generated for those files.

  • An issue has been resolved that could cause the file watcher to lose data at the boundaries when chunking was set to auto.

Security

  • Various CVEs were remediated as part of this release.

User Interface

  • Resolved the issue where the resizing of a browser window threw the error Error: Cannot read properties of undefined (reading 'getBoundingClientRect') or this.getElements()[0] is undefined.

  • Resolved an issue where certain Python libraries such as spacy, tensorflow, keras did not work in the Scratchpad.

  • Postgres and SqlServer nodes no longer incorrectly have a password field.

  • Resolved the issue where the Views page in the UI occasionally froze when performing certain operations using the map.

  • The instances where the error "Failed to compare saved assembly [assembly name] to deployed version" were erroneously triggered are now resolved.

  • Resolved the issue where resizing your browser window throws "Error: Cannot read properties of undefined (reading 'getBoundingClientRect') or this.getElements()[0] is undefined".

  • Duplicate filters are no longer being reported after creating then saving a filter.

  • Navigating to a Dashboard/Screen from Actions is now working in Views. Previously, you could receive the error "Error: Cannot read properties of undefined (reading 'navigate')" or be redirected to the homepage.

  • Creating a View with a Datagrid, copy it to a second View and pasting, and then returning to your original View would cause your changes to be lost. This has now been resolved.

  • Fixed the incorrect JSON file format that was causing issues when trying to download pipeline logs from the UI.

Third-party Dependencies

kdb Insights Enterprise 1.8.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.8.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.8.0.tgz
Operator kxi-operator-1.8.0.tgz
CLI kxicli-1.8.0-py3-none-any.whl
C interface kxi-c-sdk 1.8.0
RT Python interface kxi-rtpy-1.8.0
Java interface kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.7.0
Infrastructure kxi-terraform-1.8.0.tgz

Upgrade notes

  • Encryption of data in transit{:target="blank"} is turned on by default for fresh installs. When you upgrade _kdb Insights Enterprise the CLI gives you the option to turn encryption on or off. To change this setting after installation, perform an upgrade.

  • When creating a new database through the kdb Insights Enterprise UI, the isSharded schema property is now being defaulted to true. See here for more details on the implications of partitioned tables.

Known Issues

  • Upgrading kdb Insights Enterprise on Openshift will report that the insights-on-k8s task will fail to install Istio. This can be safely ignored. Please follow the instructions to install Istio as a prerequisite.

  • Using a Stream Processor Amazon S3 reader may fail when running on EKS. To workaround this, please set the tenant field to kxinsights.

  • If you have has an existing kxi-license secret created from a kc.lic or k4.lic, kxi install setup will re-use this secret, but will not set the appropriate values in the resulting values.yaml. See the available license types here.

  • Occasionally installs and upgrades may fail with an error when running kxi install. This can occur due to timeouts with the helm upgrade, giving errors such as the below:

        timed out waiting for the condition
        context deadline exceeded
    

    Re-run the install or upgrade to resolve this issue.

  • With entitlements enabled, the Query tab in the UI will not return data as there are issues with resolving the correct database name when checking entitlements. In order to workaround this issue, you should create a second entitlement for each database / assembly.

    For example if you create and entitle an assembly called dfx-assembly, you should also create an identical entitlement called dfx-assembly-qe. This -qe entitlement should be kept in-sync when updating the main one.

    This restriction will be removed in a future release.

  • qsql requests, made either through the Q tab (Query environment) in the UI or via REST to /qsql, do not work with entitlements. The .kxi.qsql API should be disabled if using entitlements. This is disabled by default in kdb Insights Enterprise. See here for more details.

1.7.4

Release Date

2024-01-04

Fixes

Expand to see the full detail of resolved issues here

Database

  • Improved failure recovery for object storage in Storage Manager. This resolves a case where EOI would not complete and would report SM Flushing filesystem with [directory] errors.

  • Resolved a bug in which EOIs were not completing, and NoSuchKey errors were occurring after moving data to object storage.

  • Improvements have been made to the refresh token logic for AWS deployments. Previously, token refresh logic was erroneous if the session was not requesting object storage bucket data. The refresh rotated every 30 minutes instead of at the session expiry time. Session expiry time is configured with EC2 instance metadata, or by KX_KURL_AWS_TOKEN_DURATION when using Amazon STS.

User Interface

  • Resolved an issue where certain Python libraries such as spacy, tensorflow, keras did not work in the Scratchpad.

  • In the Query tab, duplicate filters are no longer being reported after creating then saving a filter.

Security

  • Various CVEs have been remediated as part of this release.

Third-party dependencies

kdb Insights Enterprise 1.7.4 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.7.4 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.7.4.tgz
Operator kxi-operator-1.7.2.tgz
CLI kxicli-1.7.2-py3-none-any.whl
C interface kxi-c-sdk 1.7.0
Java interface kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.7.0
Infrastructure kxi-terraform-1.7.0.tgz

1.7.3

Release Date

2023-11-01

Fixes

Expand to see the full detail of resolved issues here

Database

  • Optimizations made to the getData API to improve query performance when the prtnCol is sorted.

Security

  • Various CVEs have been remediated as part of this release.

Third-party dependencies

kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.7.3.tgz
Operator kxi-operator-1.7.2.tgz
CLI kxicli-1.7.2-py3-none-any.whl
C interface kxi-c-sdk 1.7.0
Java interface kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.7.0
Infrastructure kxi-terraform-1.7.0.tgz

1.7.2

Release Date

2023-10-03

Improvements

  • SQL Queries

    • SQL queries will now use less memory if late data is enabled and you are querying data for the current day.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Resolved an issue where data for dates modified after the retention period was not getting moved from the first to the secondary HDB tier.

  • Improved the startup time for HDB DAPs, allowing for faster availability of processes that serve queries.

Stream Processor

  • The Apply Schema transform node now works with NULLs in string types.

Third-party dependencies

kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.7.2.tgz
Operator kxi-operator-1.7.0.tgz
CLI kxicli-1.7.1-py3-none-any.whl
C interface kxi-c-sdk 1.7.0
Java interface kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.7.0
Infrastructure kxi-terraform-1.7.0.tgz

1.7.1

Release Date

2023-09-19

Fixes

Expand to see the full detail of resolved issues here

Database

  • Resolved an issue with the Storage Manager not rolling back aborted transactions if an EOD failed.

  • Deleting data from the HDB using a filter option now correctly performs the delete. Previously, this returned an error claiming that the column(s) being filtered didn’t exist.

  • EOIs are no longer delayed after an EOI recovery event. Previously, it was possible for EOIs to become held up if the process was terminated then subsequently recovered. The EOIs would still complete successfully, but with a delay.

Security

  • Resolved CVE-2023-37920 Certifi's removal of e-Tugra root certificate, by upgrading the spaCy dependency.

Stream Processor

  • Attempts to horizontally scale pipelines using the direct write feature will now explicitly error since this is not currently supported.

Deployments & Upgrades

  • Resolved the known issue in 1.7.0 in which a race condition could cause an upgrade to fail with an error when running kxi install upgrade.

Third-party dependencies

kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.7.1.tgz
Operator kxi-operator-1.7.0.tgz
CLI kxicli-1.7.1-py3-none-any.whl
C interface kxi-c-sdk 1.7.0
Java interface kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.7.0
Infrastructure kxi-terraform-1.7.0.tgz

1.7.0

Release Date

2023-08-25

New Features

  • End-of-day writedown memory controls

    The Storage Manager (SM) process now supports restricting memory consumption for data writedown, giving the opportunity for cost savings. Reducing the RAM allocated to SM may cause EOD to run longer than if RAM was unrestricted.

    For guidance on tuning your RAM setup, see Performance best practices and the recommended system sizes for Azure here.

  • One-node RT

    kdb Insights Enterprise now allows you to optionally deploy an RT Stream as a one-node cluster instead of a three-node cluster when deploying via the kdb Insights CLI.

    • In a three-node cluster, RT provides fault tolerance and high availability (HA).
    • In a one-node cluster, RT provides fault tolerance, but the cluster is not highly available. This configuration saves resources and reduces costs for workload use cases that do not need to be highly available. In the event that the single RT node goes down, the publisher then continues to write to its local RT log file. When the RT node restarts, the publisher retrieves any data that is present on the publisher node that it has not yet received.

    See here for more details on RT architecture.

  • C Subscriber support in Reliable Transport

    The C language interface for RT is now able to subscribe to the data on an RT stream. See here for more details.

  • Backfill a new column with derived data

    When adding a new column to your schema, you can now define a custom function that will backfill the new column with derived data.

  • Publishing derived data from custom APIs

    Custom APIs that generate derived data can now stream and/or persist their output.

  • Appending data on batch ingest

    When ingesting data using the direct write feature, you now have the option to append data rather than replace it. See overwrite option for more details.

  • Data Entitlements for querying data Beta

    When querying via REST or IPC, kdb Insights Enterprise now provides data entitlement restrictions for limiting user access to the data contained within specific deployed databases.

  • Parquet file reader Beta

    Ingesting Parquet files from local storage or from an AWS S3 bucket using the q reader API, Python reader API and the UI is now enabled. This feature will be enabled for Azure and GCP and include a file watcher with file pattern matching in a future release.

Improvements

  • End-of-day writedown performance improvements

    The Storage Manager (SM) process has been optimized to significantly improve its performance - up to 50-70% reduction in EOD duration. This improved EOD performance can reduce the memory allocated or the time taken for writedown, depending on your workloads and configuration.

    For guidance and best practices on how to configure your schema and system to take advantage of these optimizations, review Performance best practices and the recommended system sizes for Azure here.

  • User interface

    • The database schema editor has been completely redesigned with a new responsive table grid for entering column metadata details. The editor uses keyboard driven actions for efficient data entry. Backed by HTML canvas rendering, hundreds of columns can be added with ease.

    • The Database configuration now supports different storage classes for the IDB and HDB (for example, FSx for Lustre). There is now a dropdown under Advanced Writedown settings for the File System that changes the mount storage class. See here for details.

    • The code editors in the Pipeline Editor can now be used to evaluate q and Python code. See Executable Editors in the Building a Pipeline section for more information.

    • Virtual queries are now available in Views. See Virtual data sources for more information.

    • The ‘Database Wizard’ has been removed from the UI. Databases can now be created and configured using the unified database configuration screens.

    • Query results in the UI are now shown using the timezone as returned by the query APIs for both Console and Table views. Previously, Table view would apply the user’s timezone which may have been different.

  • Readers

    • When using cloud and file readers, you can now use filename pattern matching to ingest multiple files using a single reader. This expands the ability for you to build up ingest workloads that pull in multiple files. For example, using an S3 reader, you could read all CSV files in all folders within a bucket using the pattern:

      s3://path/to/bucket/**/*.csv
      
    • When using cloud and file readers you can now use a file watcher which is usually paired with filename pattern matching described above. The reader will continually watch for new files matching the file path(s) provided. See here for details on:

      Not to be used with the Write Direct to HDB option

      The Watcher options should not be used when also choosing the Write Direct to HDB option in the Database Writer in the UI, the q or Python APIs.

    • Advanced configuration options for the Kafka Reader node in the Pipeline Builder UI allow more flexible options for security protocol configuration or changes to fetch intervals, as seen when subscribing to an Azure Event Hub Kafka connector.

  • Reliable Transport

    • Users can now launch KXI feeds from outside of their Kubernetes cluster and send data to their kdb Insights Enterprise deploy.
  • Logging

    • RT Stream logs are now included in Database logs on the Diagnostics page, and they are also included in the Database Issues counts shown on the Database Overview page.

    • When viewing a deployed pipeline, you can now click on an icon to jump straight to the pipeline logs.

  • Terraform Scripts

    • The default node types for each cloud provider have been changed to ensure that the default node group uses memory-optimised instances, and the node group for rook-ceph has storage-optimised instances:

        AWS:
    
        default node pool: "r5.xlarge"
    
        rook-ceph node pool: "i3.2xlarge"
    
        Azure:
    
        default node pool: "Standard_E4_v3"
    
        rook-ceph node pool:”Standard_L8s_v2"
    
        GCP:
    
        default node pool: "n1-highmem-4"
    
        rook-ceph node pool:”n2-standard-8"
    

    • The deployment of the nginx controller has been changed to be a Daemonset which will ensure consistent availability across all nodes.

    • The standalone installation using the terraform client scripts now enables cloud native system metrics on GKE clusters.

Fixes

Expand to see the full detail of resolved issues here

Database - When using the Filter By Label option in the Query page, you can now use more than one label for the filter. In previous releases, additional labels beyond the first were not taken into account when the query was executed.

  • Fixed an issue where schema type conversions from strings to GUIDs would fail and cause a type error.

  • Fixed an issue on certain NFS-based system (namely Lustre) where the ‘current’ database pointer could get converted to a directory resulting in an unexpected error.

EISDIR: /data/idb/current.new -> /data/idb/current
  • An error has been fixed that would occur when updating an object storage tier that used an inventory file after a data migration.

  • Fixed an issue when querying with .kxi.getData using a groupBy operation on a foreign key column where the name of the foreign key column would be thrown and the query would fail to complete.

  • Fixed an issue where renaming columns in the schema UI would incorrectly delete the underlying column and create a new one with null data. Column renames in the UI now correctly preserve the column’s data.

  • A new warning has been added to the Resource Coordinator if no Aggregators connect to it. After a wait period, if an Aggregator continues not to connect, this will be escalated to an error to indicate that the system is not ready for queries.

  • Stream resources now correctly show the amount of resources to be used when an additional stream was added. Previously the stream resources were only showing the compute settings for a single stream instead of adding both together.

  • When the kdb Insights Database was configured with an object storage tier but an inventory file was not provided, the Storage Manager would error on startup with invalid object storage URI error. This has been changed to not attempt to create an inventory file if no configuration is explicitly provided.

  • Resolved an issue when attempting to migrate HDB data to Azure Blob Storage. Previously, this would fail with a signature error.

  • Fixed a potential race condition between NFS replication and DAP reload signals. It was previously possible that a DAP may reload before NFS replicated data which could result in data gaps. A new sequencing mechanism has been added to ensure DAPs do not reload until replication is complete.

  • Values in a symbol column will now be properly available immediately after a batch ingest. Previously, these would be missing until the next EOI.

  • Renaming a sort column now longer errors.

Stream Processor

  • q code in a pipeline’s Global Code field will now be executed with the same semantics as loading a .q file. This fixes issues around expressions incorrectly being concatenated, the handling of comments, and expressions starting with .

  • The return of UDFs when generating q objects is no longer incorrectly wrapped as foreign objects. A user no longer needs to use a null python function following the UDF to unwrap it to the correct type.

Command Line Interface - Reinstalls and rollbacks no longer fail due to password history issues with the initUser.

Third-party dependencies

kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.7.0.tgz
Operator kxi-operator-1.7.0.tgz
CLI kxicli-1.7.0-py3-none-any.whl
C interface kxi-c-sdk 1.7.0
Java interface kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.7.0
Infrastructure kxi-terraform-1.7.0.tgz

Upgrade notes

  • Occasionally upgrades may fail with an error when running kxi install upgrade. This occurs due to a race condition in the 1.7.0 of the CLI when trying to replace the CRDs. The upgrade will stall and timeout at this point.

    ..
    Replacing CRD assemblies.insights.kx.com
    Replacing CRD assemblyresources.insights.kx.com
    Error: Timed out waiting for CRD assemblyresources.insights.kx.com to be deleted
    

    Re-running the upgrade should workaround the issue. This will be fixed in the next patch release.

Known Issues

  • With entitlements enabled, the Query tab in the UI will not return data as there are issues with resolving the correct database name when checking entitlements. In order to workaround this issue, you should create a second entitlement for each database / assembly.

    For example if you create and entitle an assembly called dfx-assembly, you should also create an identical entitlement called dfx-assembly-qe. This -qe entitlement should be kept in-sync when updating the main one.

    This restriction will be removed in a future release.

  • Previously cloud readers in a failure recovery scenario would begin reading file(s) from the start, even if they had read them prior to crashing. This would have caused duplicate data if using a stream or database writer. As of 1.7 this is no longer the case. One exception case remains, when using a glob pattern match in a non watching scenario. If the pipeline fails, recovers and matches a new set of files (because the underlying files in the object storage have changed) we may get duplicate data.

  • qsql requests, made either through the Q tab (query environment) in the UI or via REST to /qsql, do not work with entitlements. The .kxi.qsql API should be disabled if using entitlements.

  • Occasionally installs and upgrades may fail with an error when running kxi install. This can occur due to timeouts with the helm upgrade, giving errors such as the below:

        timed out waiting for the condition
        context deadline exceeded
    

    Re-run the install or upgrade to resolve this issue.

Backward Compatibility

If you are upgrading from a version earlier than 1.6.3 you must also read each of the relevant release notes to ensure you know all the changes that will affect the upgrade. Refer to the previous release page for older releases.

1.6.4

Release Date

2023-11-28

kdb Insights Core Version

This version of kdb Insights Enterprise runs kdb Insights Core 4.0.5.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Improved failure recovery for object storage in Storage Manager. This resolves a case where EOI would not complete and would report SM Flushing filesystem with [directory] errors.

  • Resolved a bug in which EOIs were not completing, and NoSuchKey errors were occurring after moving data to object storage.

  • Improvements have been made to the refresh token logic for AWS deployments. Previously, token refresh logic was erroneous if the session was not requesting object storage bucket data. The refresh rotated every 30 minutes instead of at the session expiry time. Session expiry time is configured with EC2 instance metadata, or by KX_KURL_AWS_TOKEN_DURATION when using Amazon STS.

  • Added a fix for the GCP system command expiry timer being discarded/ignored, which resulted in tokens not refreshing at all using gcloud auth.

Security

  • Various CVEs were remediated as part of this release.

Artifacts

type location
Enterprise insights-1.6.4.tgz
Operator kxi-operator-1.6.1.tgz
CLI kxicli-1.6.1-py3-none-any.whl
C SDK kxi-c-sdk 1.6.0
Java SDK kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.6.0
Infrastructure kxi-terraform-1.6.1.tgz

1.6.3

Release Date

2023-07-25

Improvements

  • A new quick test deploy option is available for pipelines for faster iteration during development. This deploys a pipeline in test mode to the user’s scratchpad process and is significantly faster than a full test. See quick test deploy for more information.

Fixes

Expand to see the full detail of resolved issues here

Database

  • Fixed a potential race condition between NFS replication and DAP reload signals. It was previously possible that a DAP may reload before NFS replicated data, which could result in data gaps. A new sequencing mechanism has been added to ensure DAPs do not reload until replication is complete.

  • Fixed an issue where converting a column from a string to a symbol would throw a type error.

  • An error has been fixed that would occur when updating an object storage tier that used an inventory file after a data migration.

  • A new warning has been added to the Resource Coordinator if no Aggregators connect to it. After a wait period, if an Aggregator continues not to connect, this will be escalated to an error to indicate that the system is not ready for queries.

  • REST queries could have input parameters converted to the wrong type when multiple type options were possible. REST APIs were converting to the maximum possible type instead of using the correct default type.

  • Database names in diagnostics now correctly correspond to the name of the assembly that the database maps to.

User Interface

  • Any change made on the Database UI tab will now mark the database as unsaved, and warn the user if trying to close the tab with any unsaved changes.

  • The View Errors button is now renamed to View Database Errors and points to the database error logs. To view Pipeline Errors, click on the Open Pipeline Errors icon next to each pipeline error.

  • Fixed an issue where renaming columns in the schema UI would incorrectly delete the underlying column and create a new one with null data. Column renames in the UI now correctly preserve the column’s data.

Artifacts

type location
Enterprise insights-1.6.3.tgz
Operator kxi-operator-1.6.1.tgz
CLI kxicli-1.6.1-py3-none-any.whl
C SDK kxi-c-sdk 1.6.0
Java SDK kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.6.0
Infrastructure kxi-terraform-1.6.1.tgz

1.6.2

Release Date

2023-07-03

New Features

  • A file watcher for polling cloud storage is now supported in the Stream Processor. The file watcher can monitor any Azure, AWS, or GCP object storage directory. It polls for new files to ingest into an SP pipeline. See the q reader API for details on how to configure the watch parameter in your SP reader. This feature will be available in the Python API and UI in a future release.

  • Query resiliency has been improved by adding the ability to control how processes serving queries are shut down. These processes can now be configured to stay up to serve in-progress query requests before shutting down, ensuring users still receive their data and are unimpacted by system operations. Users simply define how long a process should continue to serve requests before ultimately going offline. This is configured by setting .values.service-gateway.aggregator.aggregator.terminationGracePeriodSeconds to an appropriate duration in your values file during installation.

  • Query scalability has increased with the new ability to independently scale query processes within an assembly, allowing each assembly to have a tailored amount of these processes. Making this configurable and manually scalable lets you better size your assemblies with specific query workloads in mind. See routing configuration for more details. Furthermore, a new routing parameter called scope supports routing query requests based on your target assembly(s).

  • Reading directly from a Parquet file(s) in local storage is now supported in Stream Processor pipelines, allowing users to take advantage of this efficient common file format. See the q reader API for more details. This feature will be available in the Python API and UI in a future release.

  • Connect your existing kdb database already in object storage to your kdb Insights Enterprise system and query it.

  • Pipelines created in the UI can now be exported to be run by the CLI, either for automation or for migrating between environments. From the UI, select the pipeline and click ‘Export’. See Exporting artifacts for more details.

Fixes

Expand to see the full detail of resolved issues here

User Interface

  • In the kdb Insights Enterprise UI, new validation has been added for table names within schemas to ensure that they do not contain invalid characters. Previously, invalid characters could be included in tables names and the database would error after it was deployed.

  • In the kdb Insights Enterprise UI, you can now allow an internal stream to read data from an external source directly to a database. See stream configuration for more details.

  • When creating a new database or renaming an existing database, you are prevented from entering a database name longer than 24 characters.

Artifacts

type location
Enterprise insights-1.6.2.tgz
Operator kxi-operator-1.6.1.tgz
CLI kxicli-1.6.1-py3-none-any.whl
C SDK kxi-c-sdk 1.6.0
Java SDK kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.6.0
Infrastructure kxi-terraform-1.6.0.tgz

1.6.1

Release Date

2023-06-22

Improvements

  • In the kdb Insights Database, a new routing parameter called scope has been introduced for routing query requests based on assembly. This parameter allows the user to specifically target individual assemblies for their query.

Artifacts

type location
Enterprise insights-1.6.1.tgz
Operator kxi-operator-1.6.0.tgz
CLI kxicli-1.6.0-py3-none-any.whl
C SDK kxi-c-sdk 1.6.0
Java SDK kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.6.0
Infrastructure kxi-terraform-1.6.0.tgz

1.6.0

Release Date

2023-06-20

Improvements

  • Support for air-gapped environments has been improved and both kxi install run and kxi install upgrade support installing packaged charts (chart tgz’s) by passing the packaged chart name as an argument. If you use the default value of kx-insights for the chart repo, you can install it as follows:

    kxi install run --version 1.6.0 --filepath values.yaml insights-1.6.0.tgz
    

    This argument also supports chart references, for example:

    kxi install run --version 1.6.0 --filepath values.yaml <CHART_REPO>/insights
    
  • The kdb Insights Database can now work with or without an inventory file provided for an object storage tier. Using an inventory file for your object storage tier is still the recommended configuration, as this leverages caching and greatly improves performance when querying object storage data.

  • The kdb Insights Enterprise UI has an improved left hand menu. It can be minimized, and it is displayed alongside the Pipeline Operators list when the pipeline editor is active.

  • The kdb Insights Database now validates that primary keys for a partitioned table include the partition column. Previously, setting a primary key on a partitioned table that was not the partition column would error when queried.

  • Query timeout messages now indicate a reason for the timeout. Previously, query timeouts gave no reason, or said “timed out for unspecified reasons”. See queueing timeouts for more details.

  • Greater stability through the use of Kubernetes discovery between the Service Gateway and database processes.

  • Client-side pivot queries are now available in Views, with optional server-side computation for kdb+/q.

  • When calling kxi install delete the kdb Insights CLI now backs up assemblies before tearing them down.

Fixes

Expand to see the full detail of resolved issues here

Security

  • Various CVEs have been remediated as part of this release.

Database

  • Resiliency improvements have been made to ensure the kdb Insights Database can recover from multiple days without end-of-day processing having completed. Previously, if more than 2 EODs were missed, it was possible to get a misaligned error on recovery.

  • Validation has been added for assemblies to ensure that tiers' scheduled alignment time (snap value) are provided as the kdb+ time type (19h). Previously users could enter a timespan which would silently be ignored. Entering a timespan will now error.

Reliable Transport

  • Kubernetes service account tokens are now automatically refreshed from disk. In previous versions these were not refreshed, causing failing requests to the Kubernetes control plane and pods had to be restarted.

Deprecation

CLI Install Commands

The following three arguments have been deprecated from kxi install setup: chart-repo-name, chart-repo-username and chart-repo-url.

To prevent breaking changes the arguments will still work, but they may be removed in a future version. Adding a chart repository is now decoupled from kxi commands. The suggested approach when calling kxi install setup is to add the chart repository directly using helm repo add and to reference that repository by passing the chart reference as an argument.

If you use the default value of kx-insights for the chart repo, you can install it as follows:

kxi install run --version 1.6.0 --filepath values.yaml

If you have used a non-default chart repo value then you can install with:

kxi install run --version 1.6.0 --filepath values <CHART_REPO>/insights

CLI Lifecycle Commands

The following commands have been removed. Please use the equivalents below to manage assemblies.

removed supported command
kxi azure assembly backup kxi assembly backup
kxi azure assembly delete kxi assembly teardown
kxi azure assembly restore kxi assembly deploy

Third-party dependencies

kdb Insights Enterprise 1.6.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.6.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.6.0.tgz
Operator kxi-operator-1.6.0.tgz
CLI kxicli-1.6.0-py3-none-any.whl
C SDK kxi-c-sdk 1.6.0
Java SDK kxi-java-sdk 1.6.0
RT Bridge rt-bridge 1.6.0
Infrastructure kxi-terraform-1.6.0.tgz

Known Issues

  • In 1.6.0 only, any code evaluated in the Python editor will throw a type error. To fix the Python editor for this issue you can run the following in a q editor:

    \d .com_kx_edi
    python.run: {
        z: python.i.removeExtraIndents z;
        a: python.i.run[z; y];
        $[x;$[y;python.i.truncate (),a;a];$[y;"";::]]
        }
    
  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

    To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below

    klic license renew $LICENSE_UUID --version=latest
    
    env QLIC=$(pwd) klic license fetch $LICENSE_UUID
    

    Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.

    printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml
    
    kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml
    
  • When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. Therefore, there is a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.
  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.
  • A known issue exists within the kxi install rollback command when rolling back to prior versions. When the target version was installed with keycloak.initUser enabled. It is expected the keycloak-config-cli job will fail. This is due to the rollback trying to revert that user's password and violating the password history policy.

The rollback failure will manifest as the failure of the keycloak-config-cli job. This will create job pods as it fails and retries. In order to proceed with the rollback it's recommended to delete kubectl delete job ${RELEASE_NAME}-keycloak-config-cli.

A known issue exists during a reinstall using the CLI where the keycloak.initUser is enabled. If the user's password has changed, the reinstall will try to revert it back and be rejected by the password history policy.

This can be avoided by disabling the user import on reinstall using --import-users False as below:

kxi install run --import-users False ..

Backward Compatibility

If performing a rollback from 1.6.0 to 1.5.x, a rollback chart is required. See Upgrading for more details.

1.5.3

Release Date

2023-06-19

Improvements

  • Resiliency improvements have been made to ensure the kdb Insights Database can recover from multiple days without end-of-day processing having completed. Previously, if more than 2 EODs were missed, it was possible to get a misaligned error on recovery.

  • The kdb Insights Database can now work with or without an inventory file provided for an object storage tier. Using an inventory file for your object storage tier is still the recommended configuration, as this leverages caching and greatly improves performance when querying object storage data.

  • In cases where no data transformation is required, the kdb Insights Enterprise UI now allows Streams to be configured to write directly to the database. This reduces the amount of resources required for data ingestion and lowers operating costs.

Fixes

Expand to see the full detail of resolved issues here

Security

  • Various CVEs have been remediated as part of this release.

User Interface

  • When creating a new database or renaming an existing database, you are now prevented from entering a database name longer than 24 characters. Previously, there was no restriction which could lead to name truncation.

  • The Database Schema Code View page now accepts all schema configuration options. Previously, it would not accept blockSize or isSharded.

Reliable Transport

  • Kubernetes service account tokens are now automatically refreshed from disk. In previous versions these were not refreshed, causing failing requests to the Kubernetes control plane and a need to restart your pods to recover.

Artifacts

type location
Enterprise insights-1.5.3.tgz
Operator kxi-operator-1.5.3.tgz
CLI kxicli-1.5.0-py3-none-any.whl
C SDK kxi-c-sdk 1.5.0
Java SDK kxi-java-sdk 1.5.0
RT Bridge rt-bridge 1.5.0
Infrastructure kxi-terraform-1.5.1.tgz

1.5.2

Release Date

2023-05-31

Improvements

  • In the kdb Insights Enterprise UI, when filtering queries in the Query page, distinct is now a supported aggregation type.

  • The getData API can now perform aggregations on foreign key columns.

Fixes

Expand to see the full detail of resolved issues here
  • Kubernetes service account tokens are now automatically refreshed from disk. In previous versions these were not refreshed, causing failing requests to the Kubernetes control plane and pods had to be restarted.

Database

  • When deploying an assembly via the kdb Insights CLI, if ‘size’ is not specified for DAP instances, a default of 3 replicas will once again be applied. In 1.5.1, this default was increased to 4 to provide increased availability, but 3 has been restored as the standard to control costs. Users can configure a greater number of replicas if desired.

  • In the kdb Insights Enterprise UI, pipelines associated with a database now show in the status table. Previously, only the first pipeline would be shown.

  • In the kdb Insights Enterprise UI, the database schema overview now updates when changes are made to the database’s schema. Previously, this would only show the schema state when first viewed.

  • In the kdb Insights Enterprise UI, database documents no longer get marked as modified after a teardown or cleanup.

  • The database configuration UI now validates that the minimum rollover time is 1 minute. Previously, this could be set to 0.

Stream Processor

  • Pipelines can be now be safely torn down without checkpoints having been cleared. Previously, if a teardown was followed by a restart, it is possible the pipeline would never run.

Artifacts

type location
Enterprise insights-1.5.2.tgz
Operator kxi-operator-1.5.2.tgz
CLI kxicli-1.5.0-py3-none-any.whl
C SDK kxi-c-sdk 1.5.0
Java SDK kxi-java-sdk 1.5.0
RT Bridge rt-bridge 1.5.0
Infrastructure kxi-terraform-1.5.1.tgz

1.5.0

Release Date

2023-05-09

New Features

UI

  • The kdb Insights Enterprise UI has a new and improved look and user experience for viewing the details of your database(s). All relevant details about a database are now grouped together in a single summary page, giving users an intuitive view into the details and health of its components. Users can quickly see if any errors or warnings are present, and drill down into the details to investigate directly from the page. The details of your schema are conveniently presented for reference as well. To accompany this new interface, the term Assemblies has been simplified to Database in the UI. Assemblies remain the underlying concept tying components together, and are still referenced in documentation. Manual upgrade of old streams and schemas may be required. See upgrading entities in the UI for more details.

Queries

  • To protect against reserved word and column name conflicts, the way labels are passed to a query has changed. Labels should now be specified either as a dictionary, or with the prefix label_. The old label style is still supported in 1.5, but is deprecated and will be disabled by default in 1.6. See upgrade considerations on labels for more details.

  • Distinct has been added to the available aggregations in getData for database queries.

Pipelines

  • Users can now use Python code in all areas of the pipeline UI, allowing for further flexibility in Python user experience with kdb Insights Enterprise. Previously, the addition of Python code to sections of pipeline UI were limited in scope to the function cells of the pipeline configuration forms.

  • The Stream Processor Python API can now leverage automatic schema conversion using a new function kxi.sp.transform.schema.

  • Users can now use the Kafka Schema Registry in Python and the UI. See kxi.sp.read.from_kafka for Python, and the Kafka reader.

  • String manipulation utilities have been added to the UI and the Stream Processor Python and q APIs, allowing users to selectively convert the string data to upper or lowercase representations for the purpose of data cleansing.

  • Non-UI deployments of Stream Processors now support pipeline replicas, providing enhanced resilience to deployed pipelines. Users can now choose to run their pipeline with replicas for high availability in the Stream Processor. This replaces the previous use of pipeline groups.

Reliable Transport

  • Reliable Transport now offers subscribers the ability to use dynamic client-side filtering, enabling users to control what parts of the data feed their subscribers need to receive. This can greatly improve ingestion performance by eliminating the need for subscribers to handle discarding of unnecessary data. See Filtering for more information.

  • RT has introduced logHistory, a new API to assist in analysis and debugging. The API returns metadata about the RT log file history for logs that are no longer available. This includes:

    • The log file name.
    • The time of the first message added to that log file.
    • The position of the first message added to that log file.

    See Log File Meta Data for more information.

Command Line Interface

  • A rollback command has been added to the CLI to simplify rolling back your installed instance of kdb Insights Enterprise after upgrading.

  • A kxi user subcommand has been added to the CLI to support managing users programmatically. This provides greater flexibility than using the keycloak initUser values at deploy time.

Standalone Installation

  • Ability to control which components should be enabled (cert-manager,ingress-nginx, sharedfiles storage class). The available components are defined on the configuration file (kxi-terraform.env).

Improvements

  • Batch ingest now supports ingesting tables that have no symbols. Additionally, extra validation has been added to verify symbol integrity.

  • Batch ingest now supports sorting of columns.

  • The kdb Insights Database gateway now supports returning responses that are greater than 2GB for IPC requests. See routing configuration for more details.

  • Database schemas in the UI can now be modified for assemblies that have been stopped. See schema modification.

  • Requests that supply a log correlator (logCorr) will now see that correlator in the Service Gateway logs. Previously a random correlator was used in the Service Gateway logs.

  • Users can now utilize backtick syntax when defining aggregations in Views, e.g. `c1`avg`price.

  • Pipeline editors can now be resized vertically.

  • Labels for improved accessibility have been added to the Pipelines screen.

  • Added significant improvements for accessibility using keyboard navigation and/or screen readers in the UI.

    • Users can exit a text area in a code block on the Query page by first pressing "CTRL+M" before pressing the "Tab" key.

    • The entity tree menu button can now be opened when hovering your mouse over the entity tree. Arrows can then be used to navigate.

    • "Code View" on the Schemas page can now be closed with the "Esc" key.

    • The Views and Pipeline pages now allow draggable elements to be inserted/dragged onto the page with the use of your keyboard.

- Kubernetes 1.25 is now supported and the default version for standalone installations.

- Ensure the latest AWS CLI is installed on the docker image used for standalone installations

Fixes

Expand to see the full detail of resolved issues here

Database

  • Additional historical tiers that used object storage in the UI now allow dashes and spaces in file names. Previously they would error for dashes and spaces.

  • Migrating tiers to object storage could fail with the eventual error SMDBM Migration discontinued due to failure if multiple threads were used.

  • New validation has been added to prevent accidentally adding a object storage tier to a database that does not already have one. This would previously error at EOD with a file not found error.

  • Schema conversions are now more robust on network based file systems (ex. Rook Ceph). Previously, this could result in a EXDEV. OS Reports: Invalid cross-device link error in the Storage Manager.

  • Changes to disk sort columns in a schema that would be incompatible with the current disk sort will now error with a clear message at deploy time. Previously, this would error with a u-fail error either at an EOI or EOD.

  • Batch ingest can now gracefully handle scenarios where there is no data provided in the session. Previously, this would result in the following error: Error reporting to licensing server: Value must be a long, leaving the ingest pipeline stuck in a RUNNING state.

  • When removing a symbol column from a table, the SM should be able to properly handle its removal as well as the sym file.

  • If an in-memory sort is applied to a column in a schema, it is now sorted on arrival in the RDB. Previously, if data arrived out of order with this set, an s-fail error would be thrown.

  • In the Storage Manager metrics, the kxi_sm_hdb_date_records metric no longer reports an empty value at startup, which resulted in false alerts about the HDB being offline.

  • The Storage Manager now double buffers its state table in order to protect against corruption.

  • Rapid successive queries using qsql, such as those from Views in the kdb Insights Enterprise UI, no longer have the potential to return query results out of order.

  • The kxi_sg_pending metric would previously be decremented twice when a connection closed, resulting in negative pending values.

  • The Service Gateways were creating duplicate connections to Resource Coordinators while they were waiting to come online. This was resulting in a large number of redundant connections being opened between the services. In large deployments, this could result in conn errors in the Resource Coordinator.

  • RDBs and HDBs that require a dependency on the IDB will now error at startup if one is not specified as part of the mount configuration.

User Interface

  • Previously, a database would show a loading spinner if one of its pipelines had reached a ‘finished’ state while its other components were in a ‘ready' state. This has now been rectified and the database will display a ‘ready’ state in this case.

  • An issue with pasting into text fields in a pipeline node’s configuration has been fixed. Previously, the node may be been duplicated.

Stream Processor

  • Pipeline names are now properly displayed in the Running Pipelines section of the Overview page. Previously, these were displaying the pipeline ID and were not clickable for viewing further information.

  • Pipelines deployed via as part of a Database now properly set Kubernetes secrets. Previously, secrets were only being set properly for standalone pipelines.

  • The presence of a UTF-8 byte-order mark at the beginning of CSV files will now be automatically removed from data when acted on by the Stream Processor CSV decoder.

  • Expression readers now return the result of Python lambdas.

  • Window nodes in the Stream Processor can now accept dictionaries as input batches. Previously, incoming dictionaries would be merged with input batch in the buffer, instead of appending to a buffered list of dictionaries.

  • The python write.to_database node previously converted assembly names to booleans. This is now resolved, and also fully supports the direct write functionality.

  • The Stream Processor's Protobuf decoder no longer ignores the value of As List when a pipeline is deployed from the UI. Previously, this value was being overwritten.

  • The Stream Processors Protocol Buffer Encoder node now properly saves the selected Payload Type.

  • The Stream Processor Database Reader now supports SQL statement queries. Previously, this node had erroneously suggested the use of qsql statements for querying which was unsupported.

  • The default onFinish functions for the apply node have been updated to have the correct parameters and examples of flushing a buffer.

  • When using directWrite functionality in .qsp.write.toDatabase, table names as strings are now supported. Previously, this had to be a symbol.

  • Copy/pasting nodes within a pipeline document while other pipeline documents are open in the background no longer produces duplicates. Previously, the copy/paste operation was being applied to both the intended pipeline as well as the unintended pipelines, resulting in nodes disappearing due to shared names/ids.

  • Security vulnerabilities patched in the Stream Processor for the following CVEs: CVE-2022-41910, CVE-2022-41902, CVE-2022-41900, CVE-2022-41880.

  • Query sessions had previously set error trapping to 1 within the q session, resulting in users being unable to interact with their query session. This behaviour manifested in users being unable to interact with their query sessions until they were manually restarted.

Command Line Interface

  • With previous versions of the kdb Insights CLI, the kxi install list-versions command would return an outdated list of versions. This has now been resolved and the list will be updated on each invocation.

Standalone Installation

  • Fix Rook-Ceph OSD not getting assigned if underlying node changes

Third-party dependencies

kdb Insights Enterprise 1.5.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.5.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.5.0.tgz
Operator kxi-operator-1.5.0.tgz
CLI kxicli-1.5.0-py3-none-any.whl
C SDK kxi-c-sdk 1.5.0
Java SDK kxi-java-sdk 1.5.0
RT Bridge rt-bridge 1.5.0
Infrastructure kxi-terraform-1.5.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly from 1.3.x to any newer release, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data
  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
    Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml
    
  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
    Tearing down assembly dfx-assembly
    Are you sure you want to teardown dfx-assembly [y/N]: y
    Waiting for assembly to be torn down  [------------------------------------]    0%
    $ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
    Tearing down assembly iot-assembly
    Are you sure you want to teardown iot-assembly [y/N]: y
    Waiting for assembly to be torn down  [------------------------------------]    0%
    
  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>
    
  6. Carry out the upgrade

  7. Restore assemblies

    kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig
    

Restriction on upgrading to older versions

The CLI will not allow you to "upgrade" to an older version of kdb Insights Enterprise. Please use the rollback command to revert to a previous version.

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

    To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below

    klic license renew $LICENSE_UUID --version=latest
    
    env QLIC=$(pwd) klic license fetch $LICENSE_UUID
    

    Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.

    printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml
    
    kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml
    
  • Publishing of a single message larger than 1GB is not supported.
  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
  • When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. Therefore, there is a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.
  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

  • A known issue exists within the kxi install rollback command when rolling back to prior versions. When the target version was installed with keycloak.initUser enabled. It is expected the keycloak-config-cli job will fail. This is due to the rollback trying to revert that user's password and violating the password history policy.

The rollback failure will manifest as the failure of the keycloak-config-cli job. This will create job pods as it fails and retries. In order to proceed with the rollback it's recommended to delete kubectl delete job ${RELEASE_NAME}-keycloak-config-cli.

A known issue exists during a reinstall using the CLI where the keycloak.initUser is enabled. If the user's password has changed, the reinstall will try to revert it back and be rejected by the password history policy.

This can be avoided by disabling the user import on reinstall using --import-users False as below:

kxi install run --import-users False ..

Backward Compatibility

If you are upgrading from a version earlier than 1.5.0 you must also read each of the relevant release notes below to ensure you know all the changes that will affect the upgrade.

1.4.5

Release for kdb Insights Enterprise.

Release Date

2023-04-26

Improvements

[IMPROVEMENT]: If publishing a message larger than 1GB, the publisher will now receive a messsage too large error. Users can catch the error (need more clear detail) to avoid getting an error at runtime. This would previously cause downstream subscriber processes to run out of memory during processing.

[IMPROVEMENT]: SQL queries that filter by time but do not explicitly select the time column are now supported. This would previously error.

Fixes

[FIX]: Fixed an issue where rapid successive queries using qsql such as those from Views in the kdb Insights UI could return query results out of order.

[FIX]: Schema conversions are now more robust on network based file systems (ex. Rook Ceph). Previously, this could result in a EXDEV. OS Reports: Invalid cross-device link error in the Storage Manager.

[FIX]: Window nodes in the Stream Processor can now accept dictionaries as input batches. Previously, incoming dictionaries would be merged with input batch in the buffer, instead of appending to a buffered list of dictionaries.

Third-party dependencies

kdb Insights Enterprise 1.4 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.4.5.tgz
Operator kxi-operator-1.4.2.tgz
CLI kxicli-1.4.2-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.1.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly created using a version prior to 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data
  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
    Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml
    
  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
    Tearing down assembly dfx-assembly
    Are you sure you want to teardown dfx-assembly [y/N]: y
    Waiting for assembly to be torn down  [------------------------------------]    0%
    $ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
    Tearing down assembly iot-assembly
    Are you sure you want to teardown iot-assembly [y/N]: y
    Waiting for assembly to be torn down  [------------------------------------]    0%
    
  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>
    
  6. Carry out the upgrade

  7. Restore assemblies

    kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig
    

Licensing

A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below.

klic license renew $LICENSE_UUID --version=latest
env QLIC=$(pwd) klic license fetch $LICENSE_UUID

Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.

printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml
kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Monitoring

We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes.

If you wish to continue with the higher frequency you can take one of the following steps:

-  Update the interval field within their charts.

    ```yaml
    metrics:
      serviceMonitor:
        interval: "30s"
    ```

- Override the chart values using the global value within insights/values.yaml.

    ```yaml
    global:
      metrics:
        serviceMonitor:
          interval: "30s"
    ```

- Update your assembly yaml, to override the default.

    ```yaml
    spec:
      metrics:
        enabled: true
        serviceMonitor:
          enabled: true
          interval: 10s
    ```

- Update your default within the Assembly Resource - This is generated using the insights/values.yaml.

    ```yaml
    kxi-operator:
      config:
        sidecar:
          metrics:
            serviceMonitor:
              interval: 30s
    ```

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • Publishing of a single message larger than 1GB is not supported.
  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.
  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.

"[] Failed to compare saved assembly  to deployed version, error: type"
  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

  • When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.
  • Screen readers will not read out node information represented in a pop-up when hovered over, and may freeze when attempting to read a large amount of content.

  • Unable to navigate up and down cells and read all table content when navigating using only a keyboard.

  • The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml.
keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.4

Release for kdb Insights Enterprise.

Release Date

2023-04-20

Improvements

[IMPROVEMENT]: The kdb Insights Database now supports a REST interface for loading historical datasets into an existing database. You can now perform a historical batch ingest using a simple REST API. For more details, see batch ingestion.

[IMPROVEMENT]: Query requests that supply a log correlator (logCorr) will now see that correlator in the Service Gateway logs, allowing for easier traceability in query execution and an improved debugging experience. Previously a random correlator was used in the Service Gateway logs.

[IMPROVEMENT]: Added significant improvements for accessibility using keyboard navigation and/or screen readers in the UI.

  • Users can exit a text area in a code block on the Query page by first pressing "CTRL+M" before pressing the "Tab" key.

  • The entity tree menu button can now be opened when hovering your mouse over the entity tree. Arrows can then be used to navigate.

  • "Code View" on the Schemas page can now be closed with the "Esc" key.

  • The Views and Pipeline pages now allow draggable elements to be inserted/dragged onto the page with the use of your keyboard.

Fixes

[FIX]: Security vulnerabilities patched in the Stream Processor for the following CVEs: CVE-2022-41910, CVE-2022-41902, CVE-2022-41900, CVE-2022-41880.

[FIX]: Additional historical tiers that use object storage in the UI now allow dashes and spaces in file names. Previously they would error for dashes and spaces.

[FIX]: Migrating tiers to object storage could fail with the eventual error SMDBM Migration discontinued due to failure if multiple threads were used.

[FIX]: If an SM client cannot connect to SM after a number of retries, it will now restart the pod to try to automatically recover. This resolves connection issues in DAPs when there are network connectivity issues.

[FIX]: Reading of files from Azure Storage larger than ~1MB now successfully writes to a kdb Insights database. Previously this was erroring due to an issue in the chunking of larger files.

[FIX]: The Service Gateway was creating duplicate connections to Resource Coordinators while they were waiting to come online. This was resulting in a large number of redundant connections being opened between the services. In large deployments, this could result in connection errors in the Resource Coordinator.

[FIX]: Expression readers now return the result of Python lambdas.

[FIX]: After upgrading kdb Insights Enterprise from 1.3.x to 1.4.x, clicking a node in a previously-saved pipeline in the UI no longer causes the function definition to flicker and disappear.

[FIX]: Fixed issue with 1.4.0 kxi-terraform scripts, where Terragrunt config was incompatible and was causing deployment failures on AWS.

Third-party dependencies

kdb Insights Enterprise 1.4 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4 release of the standalone infrastructure installation scripts:target="_blank"}.

Artifacts

type location
Enterprise insights-1.4.4.tgz
Operator kxi-operator-1.4.2.tgz
CLI kxicli-1.4.2-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.1.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly created using a version prior to 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data
  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
    Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml
    
  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
    Tearing down assembly dfx-assembly
    Are you sure you want to teardown dfx-assembly [y/N]: y
    Waiting for assembly to be torn down  [------------------------------------]    0%
    $ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
    Tearing down assembly iot-assembly
    Are you sure you want to teardown iot-assembly [y/N]: y
    Waiting for assembly to be torn down  [------------------------------------]    0%
    
  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>
    
  6. Carry out the upgrade

  7. Restore assemblies

    kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig
    

Licensing

A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.

To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7. Save this to a variable $LICENSE_UUID and run the commands below.

klic license renew $LICENSE_UUID --version=latest
env QLIC=$(pwd) klic license fetch $LICENSE_UUID

Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license, set the $SECRET variable if this isn’t correct.

printf "data:\n  license: %s\n" $(base64 -w0 kx.lic) > patch.yaml
kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Monitoring

We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes.

If you wish to continue with the higher frequency you can take one of the following steps:

-  Update the interval field within their charts.

    ```yaml
    metrics:
      serviceMonitor:
        interval: "30s"
    ```

- Override the chart values using the global value within insights/values.yaml.

    ```yaml
    global:
      metrics:
        serviceMonitor:
          interval: "30s"
    ```

- Update your assembly yaml, to override the default.

    ```yaml
    spec:
      metrics:
        enabled: true
        serviceMonitor:
          enabled: true
          interval: 10s
    ```

- Update your default within the Assembly Resource - This is generated using the insights/values.yaml.

    ```yaml
    kxi-operator:
      config:
        sidecar:
          metrics:
            serviceMonitor:
              interval: 30s
    ```

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.
  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.
  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.
  • The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.
"[] Failed to compare saved assembly  to deployed version, error: type"
  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

  • When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.
  • Screen readers will not read out node information represented in a pop-up when hovered over, and may freeze when attempting to read a large amount of content.

  • Unable to navigate up and down cells and read all table content when navigating using only a keyboard.

  • The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml.
keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.3

Release Date

2023-03-31

Improvements

[IMPROVEMENT] We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes.

If you wish to continue with the higher frequency you can take one of the following steps:

-  Update the interval field within their charts.

    ```yaml
    metrics:
    serviceMonitor:
        interval: "30s"
    Assemblies
    ```
  • Update your assembly yaml, to override the default.

    ```yaml
        spec:
            metrics:
                enabled: true
                serviceMonitor:
                enabled: true
                interval: 10s
    ```
    
  • Update your default within the Assembly Resource - This is generated using the insights/values.yaml.

    ```yaml
    kxi-operator:
    config:
        sidecar:
        metrics:
            serviceMonitor:
            interval: 30s
    ```
    

Fixes

[FIX]: Service Gateway: fixed remote code injection vulnerability in Snake YAML CVE-2022-1471

[FIX]: Queries to the kdb Insights Database now have a default timeout of 60s (previously 30s). This allows for longer running queries to be issued from query clients, such as the scratchpad.

[FIX]: Additional historical tiers in the UI now allow for a retention period to be set on the HDB tier. Previously this was erroring due to a validation issue.

[FIX]: In the UI, the real-time database rollover tier now correctly checks that rollover is less than the interval tier.

[FIX]: Requests to object storage (Azure Storage, AWS S3, Google Cloud Storage) will now backoff and retry if they get a throttle request from the vendor. Jitter will be introduced to the retry to ensure that replicated processes do not all retry at the same time.

[FIX]: Setting a column type to Any now correctly uses an anymap type. This was previously reset to an unselected value.

[FIX]: The REST Server's heartbeat logs are now set to TRACE to reduce log output (previously they were logged as INFO logs).

[FIX]: Pasting into a pipeline node’s configuration no longer duplicates the node.

[FIX]: Resolved issue where some pipelines with merge nodes would error with 'missing edge'.

[FIX]: Comments can now be added to the last line of the Scratchpad without causing it to error.

[FIX]: The Scratchpad now has more memory allocated for ad-hoc queries in the kdb Insights Enterprise query panel.

[FIX]: Stream Processor file reader nodes (Azure, AWS S3, File and Google storage) would only process the first file presented even when multiple files were present.

[FIX]: Object storage tiers can now be added to a database via the kdb Insights Enterprise UI.

Third-party dependencies

kdb Insights Enterprise 1.4.3 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.3 release of the standalone infrastructure installation scripts.

Artifacts

type location

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data
  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
    Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml
    
  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
    Tearing down assembly dfx-assembly
    Are you sure you want to teardown dfx-assembly [y/N]: y
    Waiting for assembly to be torn down  [------------------------------------]    0%
    $ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
    Tearing down assembly iot-assembly
    Are you sure you want to teardown iot-assembly [y/N]: y
    Waiting for assembly to be torn down  [------------------------------------]    0%
    
  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>
    
  6. Carry out the upgrade

  7. Restore assemblies

    kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig
    

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.

"[] Failed to compare saved assembly  to deployed version, error: type"
  • Please ensure all pipelines have different names. Two pipelines with the same name may not be deploy correctly.

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml.

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.2

Release for kdb Insights Enterprise.

Release Date

2023-03-17

Fixes

[FIX] Resource utilization in the UI Scratchpad has been greatly improved. This fix along with a larger default memory allocation provide greater Scratchpad process stability.

Third-party dependencies

kdb Insights Enterprise 1.4.2 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.4.2.tgz
Operator kxi-operator-1.4.0.tgz
CLI kxicli-1.4.1-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data
  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
    Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml
    
  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
    Tearing down assembly dfx-assembly
    Are you sure you want to teardown dfx-assembly [y/N]: y
    Waiting for assembly to be torn down  [------------------------------------]    0%
    $ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
    Tearing down assembly iot-assembly
    Are you sure you want to teardown iot-assembly [y/N]: y
    Waiting for assembly to be torn down  [------------------------------------]    0%
    
  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>
    
  6. Carry out the upgrade

  7. Restore assemblies

    kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig
    

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.

"[] Failed to compare saved assembly  to deployed version, error: type"
  • When creating a pipeline the user interface does not stop you from creating it with the same name as an existing pipeline, however two pipelines with the same name may not be deploy correctly. Please ensure all pipelines have different names.

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml.

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.1

Release for kdb Insights Enterprise.

Release Date

2023-03-14

Fixes

[FIX] In 1.3.3 and 1.4.0, if the Storage Manager falls behind on EODs for more than a full day, IDB links could be written incorrectly when late data was present. If this issue is encountered, manual data changes are required to recover the database including backup of the stream logs from the date of the failure. Please contact KX Support for help with recovery.

This issue is characterized by a log statement from the Storage Manager as follows: "Unexpected condition: EOXa watermark date YYYY.MM.DD is less than HDB stream date YYYY.MM.DD, setting it to match HDB stream date". The IDB may also start reporting "No such file or directory" on ordinal partitions.

Third-party dependencies

kdb Insights Enterprise 1.4.1 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts.

Artifacts

type location
Enterprise insights-1.4.1.tgz
Operator kxi-operator-1.4.0.tgz
CLI kxicli-1.4.1-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database - this will reduce (but not eliminate) the amount of duplicate data
  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
    Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml
    
  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
    Tearing down assembly dfx-assembly
    Are you sure you want to teardown dfx-assembly [y/N]: y
    Waiting for assembly to be torn down  [------------------------------------]    0%
    $ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
    Tearing down assembly iot-assembly
    Are you sure you want to teardown iot-assembly [y/N]: y
    Waiting for assembly to be torn down  [------------------------------------]    0%
    
  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>
    
  6. Carry out the upgrade

  7. Restore assemblies

    kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig
    

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule:

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps.

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • If end-of-day processing is interrupted and remains incomplete until the next EOD period, this prevents further EODs from completing and requires manual recovery.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.

"[] Failed to compare saved assembly  to deployed version, error: type"
  • Please ensure all pipelines have different names. Two pipelines with the same name may not be deploy correctly.

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml.

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.4.0

Release for kdb Insights Enterprise.

Release Date

2023-03-03

New Features

Introducing Pythonic pipeline development in the UI, dynamic logging notifications providing direct access to root cause of underlying issues, an enhanced user experience for database configuration and additional security enhancements.

[NEW] Pipelines

  • You can now use Python in the pipeline builder in the UI to perform aggregations and analytics.

  • Ingest large batches of data directly into the historical database has been moved out of beta.

    • Use fewer resources. Memory and disk resources greatly reduced during ingest.
    • Select the Write Direct to HDB field in the Database Writer in the UI, or use the q and Python APIs.

[NEW] User Interface

  • Improvements to the observability user experience with two new icons in top right hand-side of the UI banner:

    • A notification icon shows a count of new log messages. This ensures you are notified of any new error messages wherever you are in the UI. You can also click on the icon to show more details of the message, these are displayed in the right-hand third of the screen.

    • A settings icon enables you to update the application logging level for future log records written by the kdb Insights Enterprise components.

  • You can now share the URL of a View by selecting the view on the left-hand toolbar, right-clicking and choosing the new Share button. This URL just shares the content of the View and not the toolbars (e.g. it does not show the left-hand toolbar or the Design/Preview panel at the top).

[NEW] Databases

A new version of our Database Configuration page is now available, under a beta flag, that provides an enhanced user experience. It can be toggled on and off from the Settings option in the main toolbar. We recommend that you use the existing Database Configuration age for all running databases, but please try the new user interface out and give us your feedback.

[NEW] Packages

The user interface now allows you to configure which Packages and their associated Custom APIs are used within individual databases, this had previously only been possible through kdb CLI deployments.

[NEW] Security Improvements

  • Keycloak has been updated to version 19. This impacts the following:
    • The Keycloak Admin console has been changed
    • Logout screen provides a redirection to the login page

Improvements

Keycloak realm

In this release, the handling of the Keycloak realm has been improved. This realm contains the role definitions, users and clients. Previously the realm was not being imported by default on install. The role definitions will always be imported as part of an install or upgrade now. The users and clients will only be imported by default on install (kxi install), and not on upgrade.

To make changes to the initClient and initUser on upgrade, you will need to tell the application to import them using the setting below.

keycloak:
  importUsers: true

Reliable Transport

The kdb Insights Enterprise transport layer has adopted a version of Raft called NuRaft. This improves the fault tolerance and performance of kdb Insights Enterprise. NuRaft is an open source version of Raft. Migrating to NuRaft also provides kdb Insights Enterprise with additional capabilities for future enhancements to Reliable Transport.

Third-party dependencies

kdb Insights Enterprise 1.4.0 standalone install supports the following versions of third-party dependencies:

These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts .

Artifacts

type location
Enterprise insights-1.4.0.tgz
Operator kxi-operator-1.4.0.tgz
CLI kxicli-1.4.0-py3-none-any.whl
C SDK kxi-c-sdk 1.4.0
Java SDK kxi-java-sdk 1.4.0
RT Bridge rt-bridge 1.4.0
Infrastructure kxi-terraform-1.4.0.tgz

Upgrade notes

Upgrading existing assemblies

With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.

Causes duplicate data

Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.

  1. Stop all external publishers to kdb Insights Enterprise
  2. Wait for all published messages to be available for querying from the database - this will reduce (but not eliminate) the amount of duplicate data
  3. Backup any assemblies that may be running

    $ kxi assembly backup --filepath assembly-state-file.yaml
    Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml
    
  4. Teardown assemblies named in the output of the backup command

    $ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig
    Tearing down assembly dfx-assembly
    Are you sure you want to teardown dfx-assembly [y/N]: y
    Waiting for assembly to be torn down  [------------------------------------]    0%
    $ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig
    Tearing down assembly iot-assembly
    Are you sure you want to teardown iot-assembly [y/N]: y
    Waiting for assembly to be torn down  [------------------------------------]    0%
    
  5. Delete the RT PVCs

    kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>
    
  6. Carry out the upgrade

  7. Restore assemblies

    kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig
    

User import

The initUser will be automatically imported on initial install but not on upgrade. This prevents the initUser password from being reset on upgrade.

A new value has been added to allow you to change this behavior from the default.

keycloak:
  importUsers: false

Default password policy

The default password policy has been updated to include the following rule

  • Not one of the previous 24 passwords

If you are upgrading from an earlier version and set importUsers: true, the password for initUser must be updated to a new, valid password in the values file before upgrade.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. See here for more details.

Downgrading to earlier versions

Downgrading to earlier versions was previously supported by executing kxi install upgrade. This is no longer supported. To install an earlier version, execute the rollback steps .

Schema Configuration Changes

The field updTsCol is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.

Supported Python versions in the CLI

Support for Python 3.7 has been dropped in the CLI.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • If end-of-day processing is interrupted and remains incomplete until the next EOD period, this prevents further EODs from completing and requires manual recovery.

  • Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.

  • Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.

  • Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.

When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

  • When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.

  • When writing Python code in the scratchpad, if you type from kxi import sp, it will automatically auto-correct to from kxi import super. The workaround for this issue is as follows:

    • Accept the auto-correct
    • Add a new line to the scratchpad
    • Go back up and edit super back to sp
    • Press the down arrow to go to the next line
  • If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.

"[] Failed to compare saved assembly  to deployed version, error: type"
  • When creating a pipeline the user interface does not stop you from creating it with the same name as an existing pipeline, however two pipelines with the same name may not be deploy correctly. Please ensure all pipelines have different names.

  • Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.

  • If a reader node is added to a pipeline that invokes .qsp.read.fromFile using chunking, then compilation fails.

When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col. Symbols lists are currently not supported.

The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml.

keycloak:
  postgresql:
    primary:
      resources:
        limits:
          cpu: 2000m
        requests:
          cpu: 50m

1.3.4

Release for kdb Insights Enterprise.

Release Date

2023-03-13

Fixes

[FIX] As of 1.3.3, if the Storage Manager falls behind on EODs for more than a full day, IDB links could be written incorrectly when late data was present. If this issue is encountered, manual data changes are required to recover the database including backup of the stream logs from the date of the failure. Please contact KX Support for help with recovery.

This issue is characterized by a log statement from the Storage Manager as follows: "Unexpected condition: EOXa watermark date YYYY.MM.DD is less than HDB stream date YYYY.MM.DD, setting it to match HDB stream date". The IDB may also start reporting "No such file or directory" on ordinal partitions.

Artifacts

type location
Infrastructure kxi-terraform-1.3.1.tgz
kdb Insights Enterprise insights-1.3.4.tgz
Operator kxi-operator-1.3.1.tgz
CLI kxicli-1.3.1-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0

Batch Ingest Known Issues

  • In the unlikely event of two batch ingests being done within an end of interval (EOI) event (for example, a 10 min window) a one hour delay would occur for the subsequent ingest, but there would not be any data loss, just a delay.
  • If the Storage Manager (SM) crashes during a batch ingest, the ingest will fail. On SM recovery, it will not try again and must be restarted, but no clean-up is required.

Upgrade Notes

  • When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Resource Coordinator

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" can be misleading in the timeout message. It should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

Python Scratchpad for Queries

  • If the script consists of multiple statements, the console does not print the output of the last evaluation, and instead it only displays the expression before it is evaluated.

S3 Import Wizard

  • If you select the source, S3, and click Next, no error will be thrown, which is invalid. At least one file needs to be present.

Keycloak Config CLI

We recommend that you enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default Password Policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the Insights values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

If you currently set initUser.enabled=true or initClient.enabled=true within your own values.yaml, you may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where a Keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, you must set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly blockSize Changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

  • if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
    • this has the highest performance, but has no RAM limit on received data
  • if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
    • the smaller this number is set to, the worse ingest performance but stronger RAM limits; this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

  • On startup of pods, the following error might be observed after approximately three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.
sm:
  size: 1
  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation does not rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, you will not be redirected back to the login screen.
  • If you were logged out due to inactivity, you might see a "Logout failed" error.

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}.

1.3.3

Release for kdb Insights Enterprise.

Release date

2023-01-17

New features

Functionality for improved ingestion, efficiency and reduction of resources. Take advantage of an IDE to develop your databases, pipelines and query interfaces.

[NEW] Improved ingestion

  • [Beta] Ingest large batches of data directly into the historical database:

    • Use fewer resources. Memory and disk resources greatly reduced during ingest.
    • Select the Direct Writing field in the Database Writer in the UI, or use the q and Python APIs.

    See details here for known issues.

[NEW] Writing data to Amazon S3

  • You can now write data to Amazon S3 by either:

[NEW] Improved IDE capabilities

  • Build YAML files and upload them using kdb Insights CLI:

    • Speed up development workflow, improve productivity.
    • Access to the Kubernetes control plane is no longer needed. This is now handled for you by the CLI.
    • Assembly management using CLI.
  • If you have access to KX Analyst, you can use it to develop functions. You can now:

    • Enjoy a rich IDE development environment with syntax highlighting for YAML files.
    • Query kdb Insights Enterprise data using REST API or query interface.
    • Upload using the CLI or as part of CI/CD pipeline.
    • Build a complete SDLC that harnesses the flexibility of KX Analyst against kdb Insights Enterprise data.

    Note

    Contact your KX Sales representative if you are interested in using KX Analyst. In future releases, additional IDEs will become available, including VS Code and Jupyter Notebooks.

Improvements

[NEW] Reliable Transport

  • Reliable Transport's logging default level for the sequencer and merger has been changed from DEBUG to INFO.

Fixes

[FIX] Following a hard reset of the Reliable Transport (RT), the log files in the out folder are truncated rather than deleted. This means that the replicated log files for all subscribers to the RT instance will be garbage collected once the RT archiver starts after the reset. This prevents a hard reset from causing subscriber PVCs to fill up due to old/corrupt log files remaining.

Artifacts

type location
Infrastructure kxi-terraform-1.3.1.tgz
kdb Insights Enterprise insights-1.3.3.tgz
Operator kxi-operator-1.3.1.tgz
CLI kxicli-1.3.1-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0
RT Bridge rt-bridge-1.4.0-install.sh

Known Issues

Batch ingest

  • In the unlikely event of two batch ingests being done within an end of interval (EOI) event (for example, a 10 min window) a one hour delay would occur for the subsequent ingest, but there would not be any data loss, just a delay.
  • If the Storage Manager (SM) crashes during a batch ingest, the ingest will fail. On SM recovery, it will not try again and must be restarted, but no clean-up is required.

Discovery Service

  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.

Upgrade Notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Terraform Script

Note the version. GKE cluster cannot be deployed using a 1.3.0 version of the script; a 1.3.* version of the client scripts is needed to use Kubernetes v1.22 on GKE.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the Keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled.

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

If you currently set initUser.enabled=true or initClient.enabled=true within your values.yaml, you may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where Keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly blockSize changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

  • If unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval. This has the highest performance, but has no RAM limit on received data
  • If set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM. The smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.
sm:
  size: 1
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed.

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

  • If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".

  • If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.

If the script consists of multiple statements, the console does not print the output of the last evaluation, and instead it only displays the expression before it is evaluated.

If S3 is selected as the source and Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.

1.3.2

Release for kdb Insights Enterprise.

Release Date

2022-12-05

New Features

[NEW] Pipelines

  • [Beta] Compressed files (GZIP) can now be decoded using the q and Python APIs.

  • Multiple file paths can be added to a cloud storage Reader node using the + button in the UI, or via the q and Python APIs.

  • Wildcard support using * is now available when using a single file path with the Amazon S3 Reader.

  • CSV encoding is now available, using a new node in the UI, or via the q and Python APIs.

  • It is now possible to write to Amazon S3 buckets, using a new node in the UI, or via the q and Python APIs.

  • Pipeline lifecycle events can be subscribed to via the q API. This can be used, for example, to invoke a callback once files have been processed by a pipeline.

Improvements

  • Queries will no longer time out after 60 seconds at the gateway. Longer timeouts can be configured in getData API calls using the timeout header option.

Fixes

[FIX] Query

  • An issue where DAPs could enter low-memory mode earlier than they should has been resolved.

Artifacts

type location
Infrastructure kxi-terraform-1.3.0.tgz
Platform insights-1.3.2.tgz
Operator kxi-operator-1.3.1.tgz
CLI kxicli-1.3.0-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within kdb Insights Enterprise values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

Users who currently set initUser.enabled=true or initClient.enabled=true within their own values.yaml may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, the user is required to set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly blockSize changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

  • if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
    • this has the highest performance, but has no RAM limit on received data
  • if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
    • the smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.
sm:
  size: 1
  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.3.1

Release for kdb Insights Enterprise.

Release Date

2022-11-23

Security

[FIX] kdb Insights Enterprise 1.3.1 contains security updates for a number of critical CVEs

Streams and Reliable Transport

[FIX] An issue which could cause RT archiving to fail has been fixed.

Artifacts

type location
Infrastructure kxi-terraform-1.3.0.tgz
Platform insights-1.3.1.tgz
Operator kxi-operator-1.3.1.tgz
CLI kxicli-1.3.0-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights Enterprise values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

Users who currently set initUser.enabled=true or initClient.enabled=true within their own values.yaml may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, the user is required to set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Assembly blockSize changed

The blockSize configuration within an assembly spec.tables.<table>.blockSize has been updated with the following semantics:

  • if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
    • this has the highest performance, but has no RAM limit on received data
  • if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
    • the smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced

Previously, blockSize was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize field in the assembly for each table.

Known Issues

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.
sm:
  size: 1
  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.3.0

Release for kdb Insights Enterprise.

Release Date

2022-10-26

Features

[NEW] User Interface

  • Items in the page tree have been renamed for clarity and consistency with the CLI.
    • Data is now known as Assemblies
    • Explore is now known as Queries
    • Reports is now known as Views

[NEW] Pipelines

  • An "Auto" setting for Parse Strings in the Apply Schema node has been added. The column will be parsed if the input is textual (strings or bytes), but the output type is non-textual.

  • Data types are now visible for each column for Test Deploy previews. These can be used to validate the data types and the data shape of Transforms and Maps results.

  • Data types and column names are validated for Database writers as part of a test deploy.

  • Additional Reader validation has been introduced to pipelines, to ensure correctness of settings before deploying.

  • Test deploy previews can now show non-tabular data.

  • Previews are now available for Reader nodes, before data has been decoded.

  • The visibility of a Pipeline's state has been improved in the Overview tab. Warning and error counts can be clicked on to launch a diagnostics tab for that Pipeline.

[NEW] Assemblies

  • It is now possible to see assemblies created using kxi assembly create in the UI. Note that assemblies created and managed through the command-line are not editable via the UI.

  • Additional logging has been added to the Storage Manager, to ease troubleshooting.

  • Schema configuration validation has been improved.

[NEW] Install and Packages

  • The kdb Insights Operator is no longer uninstalled by default by kxi install delete, the command line parameter --uninstall-operator must now be used.

  • Many values for kxi install setup can now be passed via command line parameter, please see the CLI Reference for more detail.

  • Improved support for rolling back to previous versions. Enter a previous version number using the upgrade command kxi install upgrade to roll back.

  • The current kubectl context is now used as the default in all commands, instead of the namespace specified in the cli-config file.

  • kdb Insights Python modules for packaging and machine learning are no longer dependent on libc.musl, and can now be used for development on Windows and Mac.

  • Custom user query APIs and Aggregation functions can be installed and initialised within Data Access Processes and Aggregators as outlined here.

  • Custom user query APIs and Aggregation functions can be installed and initialised within Data Access Processes and Aggregators as outlined here.

[NEW] Observability

  • Reliable Transport diagnostics are now available in diagnostics exports from the UI.

[NEW] Infrastructure

  • The base image for most components has been moved to Rocky Linux.

Fixes

A large number of fixes have gone into kdb Insights Enterprise 1.3.0. Known issues which have been resolved, along with some other highlights, are listed below.

[FIX] Streams and Reliable Transport

  • A hard reset can now be performed on Reliable Transport, to recover from stream corruption. Please contact KX Support for usage instructions.

[FIX] Pipelines

  • Pipelines which have their data cleared when they're torn-down can now be re-deployed with the same name. Previously, this would cause RT sequence number problems, stopping data from flowing.

  • Persistent pop-up errors in the UI after resizing a Map node have been resolved.

  • .qsp.read.fromDatabase can now be used in pipelines.

  • Renaming a pipeline node no longer disconnects its edges.

  • Empty pipelines are no longer deployable.

[FIX] User Interface

  • Validation fixes so that date and time fields can be entered without use of the picker.

[FIX] Java SDK

  • Occasional authentication failures when querying via Java SDK have been resolved. These would previously be seen more frequently on slower connections.

Artifacts

type location
Infrastructure kxi-terraform-1.3.0.tgz
Platform insights-1.3.0.tgz
Operator kxi-operator-1.3.0.tgz
CLI kxicli-1.3.0-py3-none-any.whl
ODBC Driver kodbc 1.3.0
Java SDK java-sdk 1.3.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Keycloak Config CLI

It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.

This can be enabled by setting:

keycloak:
  keycloakConfigCli:
    enabled: true

in your values file if you are deploying Keycloak as a part of KX Insights.

If you use a shared Keycloak instance, this can be enabled by setting:

keycloak-config-cli:
  enabled: true

in your values file.

Keycloak initUser password reset

Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth key.

Default password policy

As of 1.3.0 a default password policy is now being enforced.

The default policy is:

  • At least one uppercase letter
  • At least one lowercase letter
  • At least one symbol
  • At least one number
  • Minimum length of 14 characters or greater

Information about how the policy can be configured and adjusted can be found here.

If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.

Keycloak initUser

The Keycloak initUser password must satisfy the policy if it is enabled

Default Keycloak Credentials

As of 1.3.0 the Keycloak initUser and initClient credentials are no longer defaulted within the kdb Insights Enterprise values.yaml. These defaults would previously create the demoinsights user and test-publisher client on a new deployment of kdb Insights Enterprise.

Users who currently set initUser.enabled=true or initClient.enabled=true within their own values.yaml may receive the following errors at deploy time:

Keycloak initUser has been enabled
The following fields are required to be set
 .Values.keycloak.initUser.name
 .Values.keycloak.initUser.auth

Keycloak initClient has been enabled
The following fields are required to be set
 .Values.keycloak.initClient.clientId
 .Values.keycloak.initClient.clientSecret

If enabling the initUser you are required to set:

keycloak:
  initUser:
    enabled: true
    name: "initUsername"
    auth: "initUserPassword"

Where keycloak passwordPolicy has been enabled, initUser.auth must satisfy the policy requirements.

If enabling the initClient you are required to set:

keycloak:
  initClient:
    enabled: true
    clientId: "initClientID"
    clientSecret: "initClientSecret"

Internal Network LoadBalancers

As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer. These annotations restrict access to the LoadBalancers from outside the cluster.

To disable these annotations and permit access from outside the cluster, the user is required to set:

global:
    service:
        useInternalLBAnnotations: false

For additional configuration options see here.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
  • The user-defined variables & functions in the Queries tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • The Queries tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Queries tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.
sm:
  size: 1
  • If the cluster and/or resource configuration for the kxi-discovery-service is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.2.1

Release for kdb Insights Enterprise.

Release Date

2022-09-22

Fixes

kdb Insights Enterprise 1.2.1 is a security update for vulnerabilities in the included images for Keycloak/PostgreSQL.

Artifacts

type location
Infrastructure kxi-terraform-1.2.0.tgz
Platform insights-1.2.1.tgz
Operator kxi-operator-1.2.0.tgz
CLI kxicli-1.2.0-py3-none-any.whl
ODBC Driver kodbc 1.2.0
Java SDK java-sdk 1.2.0
keycloak-server keycloak-server-0.2.1.tgz

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Version 1.2.1 requires a custom storageClassName definition

The addition of support for packages introduces a new storageClassName defined as sharedfiles, this is defined by default in any deployments of kdb Insights Enterprise that rely on the KX provided Terraform scripts for deployment but must be added manually by a user as outlined here.

Alternatively, within the deployment configuration as outlined here, modify the storageClassName to an alternative supported RWX storage option for example rookcephfs.

Keycloak 18

The version of Keycloak used within kdb Insights Enterprise 1.2.0 and 1.2.1 has been upgraded from Keycloak 16 to Keycloak 18.

Please follow the Keycloak 18 migration steps to migrate your Keycloak data prior to upgrading kdb Insights Enterprise from versions older than 1.2.0.

Azure Marketplace upgrade

Upgrading from v1.2.0 Marketplace release has a known issue. Keycloak was upgraded in v1.2.1 which contained non backward compatible changes. The PostgreSQL secret format has changed, and existingSecret from values.yaml has an extra nested key. Please make sure values.yaml has the extra auth nested key, and that the secret containing PostgreSQL passwords, usually called kxi-postgresql, has dropped the postgresql prefix from the data keys.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data
  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.
sm:
  size: 1
  • Torn-down pipelines that include Database or Stream Writers cannot be directly restarted after having their resources removed. Upon redeploy, data will not be published downstream from the pipeline. A workaround is to rename the pipeline. Resources are removed by checking "Clear Pipeline State" during teardown in the UI, or by directly deleting the Kubernetes PVCs backing the pipeline. If either of these things are done, the pipeline must be renamed before relaunching it.

  • .qsp.read.fromDatabase cannot be run in a pipeline to query a kdb Insights Enterprise database. The following example code can be used as a workaround to query data (ex. reference data to be used within a pipeline).

onResponse:{[hdr;data]
if[0 <> hdr`ac;
    .sp.log.error ("Query failed host=%s, api=%s, code=%n, error=%s";hdr`gw; hdr`api; hdr`ac; hdr`ai)];
publish data
}
.qsp.onStart {
    h:hopen hsym `$.spenv.getReleaseName[],"-sg-gateway:5050";
    res: neg[h] (`.kxi.getData; (enlist `table)!(enlist `instruments); `onResponse; ()!());
}
.qsp.run .qsp.read.fromCallback[`publish] .qsp.write.toConsole[]
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can navigate re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.2.0

Release for kdb Insights Enterprise.

Release Date

2022-09-02

Features

[NEW] Install and Packages

  • Upgrade support{:target="blank"} in the kxi command line tool for easier upgrades. _Please take note of other Upgrade Notes below.

  • You must now provide a namespace when deleting installs using kxi install delete in the CLI.

  • Ability to list, add and delete custom versioned user code via packages to kdb Insights Enterprise via the CLI outlined here.

  • Support within packages for the definition of User Defined Functions (UDFs) written in Python or q for deployment to stream-processor or use in the Explore Window.

[NEW] Pipelines

  • MQTT QoS 0 Reader has been added.

  • Users can run Test deploys in the UI, which provide user feedback on issues when building or editing a pipeline. This includes visualization of the data at each node in the pipeline, and clear indications of which nodes failed.

  • CPU and Memory resources can now be configured from the User Interface to allow rightsizing of data pipelines.

  • Use of User Defined Functions as part of a pipeline.

  • Apply Schema is now available for use after the CSV decoder, so the optional schema parameters for the CSV node have been removed from the Pipelines UI. Users should take care to ensure they're parsing all fields in the Apply Schema node when it's being applied to CSV decoded data.

[NEW] Explore and Query

  • Significant speed improvements to Explore. Query environments are pre-allocated as part of Data/assemblies, and so switching between tabs, assemblies, or instances is now instant. Query environments must be enabled in the Database/Data Access screen, or in assembly configuration, for the assembly to be available in Explore.

  • Filtering and aggregation are now available on the Query tab.

  • There is no need to choose a database when using the Query or SQL options in the Explore window. Label filtering can be used to select particular databases. You can now read from tables with the same name and schema across multiple databases when using the Query or SQL tabs.

  • Import and use of User Defined Functions in pipelines and the Explore Window to allow running of custom code.

  • Console output can now be cleared by right-clicking in the Console and choosing "Clear".

  • The query section at the top of Explore tabs can now be collapsed for a larger code area.

[NEW] Databases

  • CPU and Memory resources can now be configured from the User Interface to allow rightsizing of storage and query services.

  • Redundant replicas possible for all query-path processes, enabling greater query uptime despite process, node, or availability-zone failure.

  • Ability for Data Access tiers (RDB, IDB, HDB) to share compute resources by using mountList within Assembly configuration rather than mountName.

  • Granular assembly deployment status showing the state of deploying components, and assembly deployment errors, are now surfaced in the UI. Mouse-over the assembly's status icon while it's deploying to see the breakdown.

  • Environmental variables can now be configured in the UI for Data Access and Storage when the Advanced view is enabled.

  • Code View for Schemas is now available in the UI. Users can edit (or copy/paste) schemas directly, rather than configuring through UI forms.

[NEW] Performance

  • Cluster overprovisioning support that can be configured to enable faster scaling, by reserving resources.

[NEW] Java SDK

  • Query support has been added to the Java SDK.

  • For ingestion there is an option to validate the data being ingested to ensure it is in the format that is expected by the database, before it is ingested.

  • Batch publishing is available, if required.

[NEW] Observability

  • It is now possible to filter for a displayed value (for example, a particular Container) in the Diagnostics screen. Right-click on the value and select "Filter by [...]".

[NEW] Infrastructure

  • A storage class named sharedfiles, that is backed by shared storage, is automatically created when deploying via client Terraform scripts. sharedfiles storage is used for storing Packages and required by kdb Insights Enterprise.

Fixes

A large number of fixes have gone into kdb Insights Enterprise 1.2.0. Known issues which have been resolved, along with some other highlights, are listed below.

[FIX] Pipelines

  • Pipelines no longer report an initial "Unresponsive" status in the UI when launching.

[FIX] Explore and Query

  • With the introduction of Query Environments, there is no longer an issue where PVCs are not removed upon assembly teardown due to active Explore tabs targeting the assembly.

  • SQL queries can now target splayed tables (reference data).

[FIX] Observability

  • All UI notifications now appear in the Diagnostics panel.

[FIX] UI

  • Redeploying Data (assembly) no longer adds extra nodes to the navigation tree.

Artifacts

type location
Infrastructure kxi-terraform-1.2.0.tgz
Platform insights-1.2.0.tgz
Operator kxi-operator-1.2.0.tgz
CLI kxicli-1.2.0-py3-none-any.whl
ODBC Driver kodbc 1.2.0
Java SDK java-sdk 1.2.0

Upgrade notes

Upgrading assemblies

When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.

Version 1.2.0 requires a custom storageClassName definition

The addition of support for packages introduces a new storageClassName defined as sharedfiles, this is defined by default in any deployments of kdb Insights Enterprise that rely on the KX provided Terraform scripts for deployment but must be added manually by a user as outlined here.

Alternatively, within the deployment configuration as outlined here, modify the storageClassName to an alternative supported RWX storage option for example rookcephfs.

Keycloak 18

The version of Keycloak used within kdb Insights Enterprise has been upgraded from Keycloak 16 to Keycloak 18.

Please follow the Keycloak 18 migration steps to migrate your Keycloak data prior to upgrading kdb Insights Enterprise.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data
  • Setting SM replicas (defining size greater than 1) in assembly YAML will cause writedown/storage and query problems. The size parameter for SM should always be set to 1.
sm:
  size: 1
  • Torn-down pipelines that include Database or Stream Writers cannot be directly restarted after having their resources removed. Upon redeploy, data will not be published downstream from the pipeline. A workaround is to rename the pipeline. Resources are removed by checking "Clear Pipeline State" during teardown in the UI, or by directly deleting the Kubernetes PVCs backing the pipeline. If either of these things are done, the pipeline must be renamed before relaunching it.

  • .qsp.read.fromDatabase cannot be run in a pipeline to query a kdb Insights Enterprise database. The following example code can be used as a workaround to query data (ex. reference data to be used within a pipeline).

onResponse:{[hdr;data]
if[0 <> hdr`ac;
    .sp.log.error ("Query failed host=%s, api=%s, code=%n, error=%s";hdr`gw; hdr`api; hdr`ac; hdr`ai)];
publish data
}
.qsp.onStart {
    h:hopen hsym `$.spenv.getReleaseName[],"-sg-gateway:5050";
    res: neg[h] (`.kxi.getData; (enlist `table)!(enlist `instruments); `onResponse; ()!());
}
.qsp.run .qsp.read.fromCallback[`publish] .qsp.write.toConsole[]
  • Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls

'Rollback' (re-apply resources) to the upgraded version:

helm rollback <release name> <current revision>
Note that despite the command name, this operation doesn't rollback to the initially installed version. It re-applies resources to the upgraded version.

There are two known issues with the UI logout.

  • After performing a logout action, the user will not be redirected back to the login screen.
  • In the case a user was logged out due to inactivity, a "Logout failed" error might be observed

In both of these cases, the logout has been successful and the user can navigate re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}

1.1.2

kdb Insights Enterprise 1.1.2 is a patch release that corrects a small number of observability and query issues.

Release Date

2022-07-15

Observability

[FIX] Metrics responses are all now terminated with a newline \n. Absence of the newline caused import issues with some metrics scrapers, for example Azure Monitor.

Explore and Queries

[FIX] The getData API endpoint no longer returns an error when an outputTZ is provided.

[FIX] Queries against the interval tier (idb) around the end-of-day rollover process could return zero results.

Infrastructure

[NEW] The Terraform scripts will now pass the correct platform to the Docker container when running them on Macs

[FIX] The Helm version in the Terraform scripts' Dockerfile has been pinned to a version without AWS compatibility issues.

Artifacts

type location
Infrastructure kxi-terraform-1.1.1.tgz
Platform insights-1.1.2.tgz
Operator kxi-operator-1.1.0.tgz
CLI kxicli-0.9.0-py3-none-any.whl
ODBC Driver kodbc 1.1.0
Java SDK java-sdk 1.1.0

Upgrade notes

Data loss of UI assembly definitions for versions earlier than 1.1.1

Assembly definitions built from the kdb Insights Enterprise UI in versions earlier than 1.1.1 will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.

To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace.

kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz

To restore, the user should run the following command post-install.

cat kxi-controller-data.tgz |  kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -

kdb Insights Operator Version

As the version of the kdb Insights Operator has not changed from 1.1.0 to 1.1.2, the command for deploying kdb Insights Enterprise is slightly different than in earlier releases. The older kdb Insights Operator version must be explicitly passed to the CLI.

kxi install run --filepath values.yaml --version 1.1.2 --operator-version 1.1.0

Please see the guide to Installing kdb Insights Enterprise or Upgrading kdb Insights Enterprise for more detail.

New functionality and roles

kdb Insights Enterprise 1.1.1 includes new features permissioned by additional roles. If you are upgrading to 1.1.2 from a version earlier than 1.1.1, please see the section about authentication upgrades for more information about how to incorporate realm changes, including new roles, during an upgrade.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.

  • Currently users will be logged out but will not be notified via the UI. If users consistently encounter errors, we suggest they log out and in again.

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • The drop-down list of assemblies in the first Explore tab opened after applying the first kdb Insights Enterprise assembly will take a short while to populate. Try again after a few seconds if nothing appears immediately, or close and re-open the Explore tab to see the list and select an assembly.

  • Teardown cleanups of Data (assemblies) via the UI will not fully remove PVCs if there are active Explore tabs targeting that assembly. Close these tabs or remove their underlying sandboxes via the REST API to complete the cleanup.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data

1.1.1

kdb Insights Enterprise 1.1.1 is a patch release that improves usability and stability.

Release Date

2022-06-28

Data (Assemblies)

[FIX] Clearer tooltips for Data status to ensure users can clearly identify when their assemblies are fully deployed.

[FIX] Additional validation has been added to stop re-deployment of Data (assemblies) after schema changes – schema changes on deployed assemblies are not supported in this version of kdb Insights Enterprise.

[FIX] Additional validation has been added for schemas to ensure they can be deployed successfully.

[FIX] The serviceVolume setting is now mapped correctly in Database and Stream

[FIX] A condition where Stream log archiving could fail has been resolved, ensuring that stream log files do not grow beyond the defined limits.

[FIX] Navigation tree issues where schemas could not be opened and deletion failures were not correctly reflected are both resolved.

Pipelines

[FIX] Nodes requiring passwords no longer store them in the pipeline configuration, they must be entered during pipeline deployment.

[FIX] Mandatory fields in nodes are now enforced in pipelines before deployment.

[FIX] Pipeline re-deployments can now optionally clear state (destroying previously persisted pipeline data, but not data already in the database) to allow edits to be made and the pipeline to be restarted from scratch.

[FIX] The error Cannot read properties of undefined (reading '0') which could be thrown when opening multiple pipeline tabs has been resolved.

[FIX] An issue where long code lines in Map nodes could cause browsers to hang has been resolved.

Explore and Queries

[FIX] Session errors Worker has been closed and r.map are no longer thrown.

[FIX] The error No aggregator available when changing assemblies in an Explore session is no longer thrown.

[FIX] Reference data joins which do not include the foreign column or where the foreign column has a different name are now possible.

[FIX] An issue where certain queries’ WHERE clause would work via REST but fail via Explore has been resolved.

[FIX] An issue where certain queries using count could throw the error Error: noupdate: `.Q `pn has been resolved.

Artifacts

type location
Infrastructure kxi-terraform-1.1.0.tgz
Platform insights-1.1.1.tgz
Operator kxi-operator-1.1.0.tgz
CLI kxicli-0.9.0-py3-none-any.whl
ODBC Driver kodbc 1.1.0
Java SDK java-sdk 1.1.0

Upgrade notes

Data loss of UI assembly definitions

Assembly definitions built from the kdb Insights Enterprise UI will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.

To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace.

kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz

To restore, the user should run the following command post-install.

cat kxi-controller-data.tgz |  kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -

kdb Insights Operator Version

As the version of the kdb Insights Operator has not changed from 1.1.0 to 1.1.1, the command for deploying kdb Insights Enterprise is slightly different than in earlier releases. The older kdb Insights Operator version must be explicitly passed to the CLI.

kxi install run --filepath values.yaml --version 1.1.1 --operator-version 1.1.0

Please see the guide to Installing kdb Insights Enterprise or Upgrading kdb Insights Enterprise for more detail.

New functionality and roles

kdb Insights Enterprise 1.1.1 includes new features permissioned by additional roles. Please see the section about authentication upgrades for more information about how to incorporate realm changes, including new roles, during an upgrade.

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.

  • Currently users will be logged out but will not be notified via the UI. If users consistently encounter errors, we suggest they log out and in again.

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • The drop-down list of assemblies in the first Explore tab opened after applying the first Insights assembly will take a short while to populate. Try again after a few seconds if nothing appears immediately, or close and re-open the Explore tab to see the list and select an assembly.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

2022-05-24 20:30:48.103035      INFO    1       Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821      INFO    1       Process 41697 terminated
2022-05-24 20:31:48.128196      INFO    1       Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002      INFO    1       Process 41714 terminated
2022-05-24 20:36:12.700467      INFO    1       Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286      INFO    0       New configuration fetched
2022-05-25 05:04:50.092259      WARN    0       Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686      ERROR   0       Config is too old
2022-05-25 05:05:50.046813      INFO    0       Waiting for the driver to sync the data

Backward Compatibility

Please see the release notes for kdb Insights Enterprise 1.1.0 if you're upgrading from an earlier version, for notes about:

1.1.0

Release for kdb Insights Enterprise.

Release Date

2022-06-07

Features

[NEW] Storage and Query

  • Late and out-of-order data handling
    • Data received out of order is recorded with an appropriate ingest timestamp
    • Late data is recorded, irrespective of where its correct destination is in the lifecycle (RDB, IDB, HDB)
    • Late data is included by default for all requests, q (getData) and ANSI SQL
    • Late data can be filtered out (for example, to see only data present in the system at a particular point in time)
    • Performance is not significantly impacted by late data arrival, including for object storage tiers
  • Reference data can be queried in the Explore Window, in a data pipeline and as part of the getData() API
  • Support for non time-series reference data in Database
  • Data in reference tables can be joined to time-series tables in free-form q queries
  • Improved SQL support for queries (joins and complex queries, except for reference tables)
  • Compression support, including object storage
  • Improved resiliency and performance

[NEW] User Interface

  • Scratchpads

    • Data can be pulled into the scratchpad using either sql or getData()
    • Freeform q, SQL or Python code can be execute on that data to further filter, aggregate and join data to explore specific records
    • Python/q functions or a model can be applied to that data to gain insights
    • A model can be fitted using that data
  • Machine Learning plugins can be added to a Pipeline created from the user interface

  • More intuitive creation of data ingest pipelines
  • Observability embedded into the UI (events and logs tab)
  • Log forwarding to each providers' log aggregator is now enabled by default in the Terraform scripts.

[NEW] Reports

  • Visualization is available for ingested data using the getData() API

[NEW] Observability

  • View Database and Pipeline logs and events from the UI
  • Ability to Export Diagnostics, including the logs, events and state for further diagnosis
  • Further integration with cloud monitoring tools
  • Additional performance metrics collected

[NEW] Reliable Transport Stream

  • Subscribers have the option to override the detail truncation options for the stream logs that they have processed
  • Improved recovery and performance

Artifacts

type location
Infrastructure kxi-terraform-1.1.0.tgz
Platform insights-1.1.0.tgz
Operator kxi-operator-1.1.0.tgz
CLI kxicli-0.9.0-py3-none-any.whl
ODBC Driver kodbc 1.1.0
Java SDK java-sdk 1.1.0

Upgrade notes

Data loss of UI assembly definition

Data loss of UI assembly definitions

Assembly definitions built from the kdb Insights Enterpirse UI will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.

To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace.

kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz

To restore, the user should run the following command post-install.

cat kxi-controller-data.tgz |  kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -

Known Issues

  • If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
   {"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
  • getData does not support connecting reference data column names that do not match. If the name of a column does not match the name of the column it is a foreign key for, then the getData reference join will fail. Example: The trade table has a column sym which is a foreign key to the msym column in the table market. The join will run into an issue since the column names are different.
  • The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)

  • Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.

  • Passwords used for authenticating to third-party services are currently stored as part of the pipeline configuration. This will be corrected in the next release. It is recommended that instead of using a password directly that a Kubernetes secret is used whenever possible.

  • Currently users will be logged out but will not be notified via the UI. If users consistently encounter errors, we suggest they log out and in again.

  • The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*

  • The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.

  • On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up no acct for 3x period, exiting. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault.

  • On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes unable to flush accounting logs. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.

  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When either the Java SDK or ODBC driver is being used, if kdb Insights Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.

Sample extract from the application log:

``` 2022-05-24 20:30:48.103035 INFO 1 Started replicator to :34.142.40.180:5000, pid 41697 2022-05-24 20:31:48.127821 INFO 1 Process 41697 terminated 2022-05-24 20:31:48.128196 INFO 1 Started replicator to :34.142.40.180:5000, pid 41714 2022-05-24 20:36:12.700002 INFO 1 Process 41714 terminated 2022-05-24 20:36:12.700467 INFO 1 Started replicator to :34.142.40.180:5000, pid 41772

e set for the deployment. 2022-05-25 05:04:50.077286 INFO 0 New configuration fetched 2022-05-25 05:04:50.092259 WARN 0 Config parse error at offset 0: Expected opening bracket 2022-05-25 05:05:50.046686 ERROR 0 Config is too old 2022-05-25 05:05:50.046813 INFO 0 Waiting for the driver to sync the data ```

Backward Compatibility

  • If you're upgrading from a previous version or the CRD is already installed, helm will not update the definition automatically. See Operator Overview Helm Warning for details on how to rectify this.

License changes

From 1.1.0 kdb Insights has moved to consumption-based licensing using a new license file, kx.lic. The user should generate the license themselves using the klic self-service tool. The documentation for this is available here.

After following the klic docs, a new kx.lic should be available for your cluster.

Then you can upload it as a Kubernetes secret using the following command.

kubectl create secret generic kxi-license --from-file=license=kx.lic

Alternatively delete your existing license secret and re-run the CLI install (kxi install ..), choosing the N when prompted if you have an existing license secret. Assuming the license secret is called kxi-license, the command below will delete it.

kubectl delete secret kxi-license

To revert to the pre-1.1.0 behaviour, the following Helm values should be set for the deployment. For details on this file see here.

global:
  license:
    ..
    onDemand: true
    asFile: false

kxi-acc-svc:
  enabled: false

Single pod Storage Manager

By default, when upgrading to 1.1.0 the Storage Manager will no longer create independent containers for the various processes used to manage data persistence. These processes now all run under a single container as child processes. As part of the upgrade, it is optionally encouraged that any assembly data workflows deployed should be upgraded to align any resources with this change.

The change requires the user to remove resources configuration for eoi, eod and dbm sections of an assembly and update the sm.k8sPolicy with a resources configuration for all these processes which will run as a single container within the sm pod.

For example

  sm:
    ..
    k8sPolicy:
      resources:
        requests:
          memory: "14Gi"
          cpu: "2000m"
        limits:
          memory: "14Gi"
          cpu: "4000m"

However if no actions are taken, kdb Insights will remain backwards compatible with the existing configuration and will use the sum of all CPU and memory resources under the sm element, using this sum to populate the sm.k8sPolicy.resources section under the sm element.

1.0.0

Release for kdb Insights Enterprise.

Release Date

2022-03-14

Features

[NEW] UI allows storage and reload of data assemblies.

[NEW] Schemas can be created and deleted in the kdb Insights Enterprise user interface.

[NEW] Assemblies can be created, saved and deployed in the kdb Insights Enterprise user interface.

[NEW] Ability to Export Assembly Definitions.

[NEW] Integration with the new License Management system. Due to these changes, existing licenses will need to be re-issued. Please contact your KX representative for more information.

Artifacts

type location
Infrastructure kxi-terraform-1.0.0.tgz
Platform insights-1.0.0.tgz
CLI kxicli-0.4.0-py3-none-any.whl
Operator kxi-operator
ODBC Driver kodbc 1.0.0
Java SDK java-sdk 1.0.0

Known Issues

  • The Service Gateway may repeatably print a NullPointerException under certain conditions. If this should occur the Kubernetes pod that the service gateway process is running on should be bounced.

  • An OOM event may be triggered when repeatedly bombarded with requests that return a large result set. In the event where an OOM occurs the client will not receive a response; the request may be retried and serviced by other gateways.

  • When choosing an assembly in the Explore Window a ‘500 Internal Server Error’ can be returned if no labels are included in the labels section of the assembly definition.

  • In the prospective menu (the customizable navigation toolbar) not all options available in the menu selector are currently supported, we recommend keeping the default menu options. However, all options are included in the Entity Tree shown when the Home menu is selected.

  • When exporting an Assembly definition from the kdb Insights Enterprise UI the file requires some cleaning before it can be imported into another environment.

    • Remove id entry from tables.
    • Remove apiVersion entry from pipelines and sequencers.
    • Remove line breaks from pipeline specs.
  • When saving a pipeline in the UI on AWS or Azure you need to change the Storage Class from “standard” (which is the default for GCP) to a storage class which exists on AWS or Azure. For example, to “gp2" for AWS, or "default" for Azure. This is done in the Persistence tab when saving. Note, the Storage Class is set separately for the controller and worker. Both must be set correctly. Scroll down to set the worker Storage Class too. Otherwise, the pipeline pod and pvc deployment will get stuck in the “pending” state as no “standard” storage class can be found. Alternatively, create a “standard” storage class per the link storage class resource.

  • Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.

  • Not all stream log files written by the Reliable Transport mechanism (RT) are deleted once the stream has been replicated to the subscriber as the RT stream log archival rules are not being applied to all processes. This will cause the disk space of certain volumes associated with the kdb Insights Enterprise cluster to fill up over time. The more data that is sent through the system the faster this will occur. This may result in out of disk space issues at the producer/subscriber and in the raft cluster. We recommend that you monitor the disk usage and manually delete old stream logs regularly to prevent the disks from filling up. A tool that monitors the disk space could be used to alert you when this intervention is required (for example: Google's Cloud Watch on GCP, or Prometheus). See the section RT Logs for locations of the stream log files and how to delete old stream logs that are not longer required.

  • If a publisher is running and either the out directory on any of your 3 nodes in the RT cluster stops growing, or your Stream Processor is not receiving data, you should restart your publisher. To investigate the progress of messages in the RT cluster stream logs and look at the RT application logs see the section RT Logs.

  • If a user leaves trailing spaces after CLI fields, such as secrets, then the trailing spaces are preserved in the values.yaml file leading to errors on deployment. For example, kxi-certificate in below yaml:
global:
caIssuer:
name: 'kxi-certificate '
secretName: 'kxi-certificate '
  • The CLI error handling is currently limited. If necessary use the underlying kubectl command to see the issue more clearly. For example, to see a yaml syntax error returned by a modified sdk_sample_assembly.yaml file, run: kubectl apply -f sdk_sample_assembly.yaml.
  • The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.

  • When using the Sample Java program, provided as part of the Java SDK, replace valFloat:float with valFloat:double in the first line of the sample.csv file before running the program.

  • The Controller expects you to call your idb, myidb, under instances if you set mounts to use different keynames than your instances an error occurs.
dap:
  instances:
    idb:
      size: 1
      mountName: myidb
      gwAssembly: insights
  • Enabling .z handler metrics accumulates memory over time and may cause containers to restart if they hit their memory resource limit.

Backward Compatibility

  • If you're upgrading from a previous version or the CRD is already installed, helm will not update the definition automatically. See Operator Overview Helm Warning for details on how to rectify this.

0.9.0

Beta 1 Release for kdb Insights Enterprise.

Release Date

2021-12-09

Features

[NEW] Run Python and q Machine Learning Models.

  • Users can add a q or Python based model to the ML registry.

  • Users can use q to retrieve either a q or a Python based model from the ML registry.

  • Users can deploy a Stream Processor pipeline that includes ML models from the ML Registry.

[NEW] Run Python functions on streaming data. Users can add Python functions into a Stream Processor pipeline to apply transformations (for example: windowing and filtering) to the streaming data.

[NEW] Support for the Kafka Schema Registry (dynamic schema updates are not yet supported).

[NEW] Users have access a single OpenAPI document. This provides documentation for the following endpoints:

  • Service Gateway

  • Client Controller

  • Information Service

  • Stream Processor

All available operations and authentication methods are documented.

[NEW] The User interface supports using ANSI SQL queries on a single database. These queries are made available via a sandbox to ensure the live database is not modified by the queries.

[NEW] Command line interface for:

  • Installation setup.

  • Enrolling and removing clients.

  • Creating and deleting assemblies.

  • Querying data.

[NEW] Login screens include KX themes

Artifacts

type location
infrastructure https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-terraform/0.1.11/kxi-terraform-0.1.11.tgz
installation https://nexus.dl.kx.com/repository/kx-insights-charts/insights-0.9.0.tgz
cli https://nexus.dl.kx.com/repository/kxi/packages/kxicli/0.1.3/kxicli-0.1.3-py3-none-any.whl

Known Issues

  • Creating a pipeline in the UI that contains an Expression node must be done through the Pipeline editor, the Import screen does not allow correct configuration of an Expression node.

  • If one of the nodes in an RT sequencer is killed or restarts, data may no longer flow through that RT. If the issue occurs on north and south sequencer, it may be sufficient to restart the assembly. Otherwise, the cluster needs to be restarted.

  • RT nodes can under certain conditions consume excessive CPU causing the cluster CPU go up to >95%. If the issue occurs open a shell to the affected node(s) and kill all the pull_server processes. They will be automatically restarted and the CPU usage will drop.

  • RT occasionally reports an inotify error. This is a resource exhaustion issue. If the issue occurs on north and south sequencer, it may be sufficient to restart the assembly. Otherwise, the cluster needs to be restarted.

  • Both SQL and q queries work with assemblies. However, when running a query against a html5evalcongroup process in the UI, SQL is not supported and q must be used instead.

  • Programmatic OpenAPI kdb Insights client generation currently fails due to an issue with the Stream Processor.

Backward Compatibility

  • If you're upgrading from a previous version or the CRD is already installed, helm will not update the definition automatically. See Operator Overview Helm Warning for details on how to rectify this.

0.8.0

Initial Alpha Release for kdb Insights Enterprise.

Release Date

2021-10-27

Features

[NEW] Secure login & authentication using Keycloak, enabling enrollment and removal of client publishers.

[NEW] Role Based Access Control with roles for pipeline viewing and editing, access to the user interface and the ability to query data.

[NEW] Licensing is via a base install key.

[NEW] Data ingestion via:

  • Publishing from the RT command line interface.

  • Reading from Kafka.

  • Publishing using direct qIPC.

  • Batch ingestion from S3 buckets.

[NEW] The User interface supports creation of stream processor pipelines; including data ingestion and transformation, and presentation in grids/charts. For this release, pipelines can be published, but drafts are only saved to the user’s browser - not to kdb Insights Enterprise.

[NEW] Data transformations can use q

[NEW] Data transformations can use a limited Python API (built on embedPy)

[NEW] Data can be streamed out via Kafka.

[NEW] Data can be visualized using the kdb Insights user interface

[NEW] Data stored in the kdb databases can be accessed via:

  • The user interface using q

  • A REST API

[NEW] Application Metrics can be generated which can be monitored via Prometheus and Grafana.

Artifacts

type location
infrastructure https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-terraform/0.1.7/kxi-terraform-0.1.7.tgz
installation https://nexus.dl.kx.com/repository/kx-insights-charts/insights-0.8.0.tgz

Known Issues

  • On AWS - deployment of infrastructure needs to be targeted at a region that has 3 availability zones - see https://aws.amazon.com/about-aws/global-infrastructure/regions_az/

  • On AWS(us-east-2) - Due to an issue specific to us-east-2 terraform tries to fetch ec2 information by hitting ec2.us-east-2.amazonaws.com normally this url should return a single public ip address but within an ec2 instance on us-east-2 region it returns 3 private ips which are not accessible via https. For this reason the application should not be deployed to us-east-2.

0.10.0

Beta 2 Release for kdb Insights Enterprise.

Release Date

2022-02-04

Features

[NEW] Support for the Kafka Schema Registry has been extended to include dynamic schema updates.

[NEW] Defining ML models can be done using Python.

  • Users can use Python to retrieve either a q or a Python based model from the ML registry.

  • Users can use Python to save either a q or a Python based model from the ML registry.

[NEW] The ANSI SQL support has been extended to include distributed databases. Currently joins are not supported and tables with the same names in different databases MUST have the same columns and properties. As in the previous release these queries are made available via a sandbox to ensure the live database is not modified by the queries.

[NEW] REST API getData function has been enhanced to include:

  • Basic aggregation functions on individual columns (for example: min, max, avg, stdev)
  • Racking and Filling
  • Filtering
  • Time range options, including temporaility

[NEW] Schemas, Streams and Databases can be created and deleted in the kdb Insights Enterprise user interface.

[NEW] KX ODBC driver for data ingestion. Enabling data ingestion by applications that support ODBC.

[NEW] KX Java SDK for data ingestion.

Artifacts

type location
infrastructure kxi-terraform-0.1.12.tgz
installation insights-0.10.0.tgz
cli kxicli-0.2.0-py3-none-any.whl
operator kxi-operator-0.2.0-rc.19.tgz
ODBC driver RPM package kodbc-0.1.52-rhel.x86_64.rpm
ODBC driver Debian kodbc-0.1.52.deb
ODBC driver Windows executable kodbc-0.1.52.exe
ODBC driver Windows msi kodbc-0.1.52.msi
ODBC driver sample program kodbc-0.1.52.zip
Java SDK rt_client-1.0.0rc6.jar
Java SDK sample program rtdemo-1.0.0rc6-all.jar
Java SDK sample csv file sample.csv

Known Issues

  • The sample assembly yaml files provided specify the same subTopic: "data". To start multiple assemblies in parallel, change the subTopic to a unique value, such as subTopic: "odbc-assembly". The corresponding change will also need to be made when enrolling the client. For example: curl https://${INSIGHTS_HOSTNAME}/clientcontroller/enrol -H "Authorization: Bearer " -d '{ "name": "my-client", "topics": { "insert": "odbc-assembly", "query": "requests" }}'

  • The UI may produce pop-up error messages in various scenarios.

  • The UI does not have Reports available.

  • kdb Insights CLI error handling is currently basic. If the CLI throws a stack trace, use the underlying kubectl command to see the issue more clearly. For example, to see a yaml syntax error returned