kdb Insights Enterprise - Previous Release Notes
This page outlines release notes from past releases. For the most recent release, refer to the latest release page.
1.9.4
Release Date
2024-06-14
Fixes
Expand to see the full detail of resolved issues here
Security
- Multiple CVEs have been resolved as part of this patch.
Cloud Readers
-
When administering a pipeline, a cloud reader node (s3, ms, and gs) could prompt you that there were unsaved changes even when no changes have been made.
-
The Azure reader file watcher was returning an error that one or more files did not exist despite the file(s) being present.
Reliable Transport
- RT's replicator was emitting an
Invalid publishing directory, was it deleted by mistake? Please enrol a different client and restart the SDK with the new client
error after recovering from a power outage of a q publisher server. This has been resolved to handle the recovery cleanly after such an outage.
UI
-
On the Pipeline page, an intermittent issue could cause performance degradation or unresponsiveness in the UI for extended periods.
-
When configuring a pipeline in the UI, setting additional
workerThreads
was not correctly applying the specified configuration. However, setting the environment variableKXI_DEFAULT_SP_WORKER_THREADS
did work. This has been resolved so either method can be used. -
The default setting for retries for database reader/writers in the UI did not match the backend defaults. This has been corrected to be consistent, with the default setting of 60 seconds for retries, and a retry wait time of 3 seconds.
-
When performing a rollback, the CLI was deploying the rollback task with a hard-coded timeout. This has been changed to a configurable value for use when rollbacks are expected to take longer.
-
When deploying a database, the Deploy popup was displaying totals that did not match the selected CPU, RAM and disk settings. The same issue was present in the Resource Summary box on the main page. These calculation errors have been resolved to display correct totals.
Database
- For a new system with an empty database, queries would not work after the first end-of-day data writedown if no data had been ingested yet.
Third-party Dependencies
kdb Insights Enterprise 1.9.4 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.9.1 release of the standalone infrastructure installation scripts.
1.9.3
Release Date
2024-05-21
Fixes
Expand to see the full detail of resolved issues here
Reliable Transport
- RT's replicator was emitting an
Invalid publishing directory, was it deleted by mistake? Please enrol a different client and restart the SDK with the new client
error after recovering from a power outage of a q publisher server. An intermittent issue with truncated log files when the publisher restarts has been resolved.
Third-party Dependencies
kdb Insights Enterprise 1.9.3 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.9.1 release of the standalone infrastructure installation scripts.
Artifacts
KX Nexus
The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.
type | Nexus location | Downloads Portal location |
---|---|---|
Enterprise | insights-1.9.3.tgz | insights-1.9.3.tgz |
Operator | kxi-operator-1.9.1.tgz | kxi-operator-1.9.1.tgz |
CLI | kxicli-1.9.1-py3-none-any.whl | kxicli-1.9.1-py3-none-any.whl |
RT C interface | kxi-c-sdk 1.9.0 | kxi-c-sdk 1.9.0 |
RT Python interface | kxi-rtpy-1.9.0 | kxi-rtpy-1.9.0 |
RT Java interface | kxi-java-sdk 1.9.0 | kxi-java-sdk 1.9.0 |
RT Bridge | rt-bridge 1.7.0 | rt-bridge 1.7.0 |
Infrastructure | kxi-terraform-1.9.1.tgz | kxi-terraform-1.9.1.tgz |
kxi-management-service | kxi-management-service-0.2.1.tgz | kxi-management-service-0.2.1.tgz |
insights-on-k8s | insights-on-k8s-0.0.16.tgz | insights-on-k8s-0.0.16.tgz |
1.9.2
Release Date
2024-05-07
Fixes
Expand to see the full detail of resolved issues here
Reliable Transport
- The RT client was failing to recognize a new log file, putting downstream subscribers into a hung state. This has been resolved.
Third-party Dependencies
kdb Insights Enterprise 1.9.2 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.9.1 release of the standalone infrastructure installation scripts.
Artifacts
KX Nexus
The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.
type | Nexus location | Downloads Portal location |
---|---|---|
Enterprise | insights-1.9.2.tgz | insights-1.9.2.tgz |
Operator | kxi-operator-1.9.1.tgz | kxi-operator-1.9.1.tgz |
CLI | kxicli-1.9.0-py3-none-any.whl | kxicli-1.9.0-py3-none-any.whl |
RT C interface | kxi-c-sdk 1.9.0 | kxi-c-sdk 1.9.0 |
RT Python interface | kxi-rtpy-1.9.0 | kxi-rtpy-1.9.0 |
RT Java interface | kxi-java-sdk 1.9.0 | kxi-java-sdk 1.9.0 |
RT Bridge | rt-bridge 1.7.0 | rt-bridge 1.7.0 |
Infrastructure | kxi-terraform-1.9.1.tgz | kxi-terraform-1.9.1.tgz |
1.9.1
Release Date
2024-04-30
Improvements
-
The
fromParquet
Reader is now supported for AWS-based deployments. For more information, refer to Readers. -
Streams can now be removed from packages using the
rm
command from the CLI. Runkxi package rm --help
in the CLI for details.
Fixes
Expand to see the full detail of resolved issues here
Database
- Client requests using compressed IPC messages are now properly serviced. Previously, if the first request from a client was using compressed IPC, that and any future requests would fail.
Terraform Scripts
- Previously, in the kdb Insights Enterprise Terraform sample client scripts, the
package-manager-migration
command failed when used in Azure deployments using the sharefiles storage class. This has been resolved. New sample scripts are available as part of this release. See the Artifacts section below.
Stream Processor
- Pipelines now throw an error when writing to a non-existent table. Previously, no errors were present in the pipeline or database logging. This was most commonly experienced when renaming a table in the database without updating the pipeline as well.
Reliable Transport
- Reliable Transport pods now use Kubernetes liveness probes to automatically bounce pods if unresponsive. This resolves a known issue in 1.9.0 in which RT pods were required to be manually restarted if an upgrade erroneously left them in a non-ready state.
Packaging
-
The UI now displays the running version of a package as opposed to the latest package that may be available.
-
subTopic
has been added as a mandatory field when adding external streams to a package. Previously, this was not present and caused a package to be undeployable. -
In the UI, it is no longer possible to create pipelines with the duplicate names.
-
The CLI now errors when trying to create and save a package that contains duplicate UUIDs, or when trying to save entities within a package that have duplicate UUIDs.
UI
- Resource allocations are no longer mandatory input fields on the Resources tab. It is now possible to leave them empty and submit your resource configuration. For more information, refer to Adjust resources.
Third-party Dependencies
kdb Insights Enterprise 1.9.1 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.9.1 release of the standalone infrastructure installation scripts.
Artifacts
KX Nexus
The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.
type | Nexus location | Downloads Portal location |
---|---|---|
Enterprise | insights-1.9.1.tgz | insights-1.9.1.tgz |
Operator | kxi-operator-1.9.1.tgz | kxi-operator-1.9.1.tgz |
CLI | kxicli-1.9.0-py3-none-any.whl | kxicli-1.9.0-py3-none-any.whl |
RT C interface | kxi-c-sdk 1.9.0 | kxi-c-sdk 1.9.0 |
RT Python interface | kxi-rtpy-1.9.0 | kxi-rtpy-1.9.0 |
RT Java interface | kxi-java-sdk 1.9.0 | kxi-java-sdk 1.9.0 |
RT Bridge | rt-bridge 1.7.0 | rt-bridge 1.7.0 |
Infrastructure | kxi-terraform-1.9.1.tgz | kxi-terraform-1.9.1.tgz |
Upgrade Notes
Upgrading to 1.9.x
-
Before you upgrade to 1.9.x, if you have any running databases, it is advised to manually stop them before upgrading, otherwise you may become unable to edit them in the UI. If this happens, you can recover the editable state by considering the following options:
- If possible, back up the running assembly yaml file with
kubectl get asm
- Tear down the running assembly manually with
kubectl delete asm
- Optional: use the
kxi package deploy
command in the CLI, specifying the name and version of the package. The package should appear underkxi package remote-list
. Refer to Package deployment for more information.
If the package doesn't reappear as editable after tearing down, you need to use the KXI CLI to perform the following operations:
- Convert the assembly yaml to a package. Refer to Converting assemblies to packages.
- Upload the package to kdb Insights Enterprise. Refer to Uploading a package.
- Deploy the package from the UI. Refer to Package deployment.
- If possible, back up the running assembly yaml file with
Known Issues
Expand to see the list of known issues here
Custom Tier Names
- Existing assemblies that have non-default tier names (i.e. names other than "rdb","idb","hdb") that are converted to packages will need to be updated to use those defaults. Presently, the UI expects the default names and is unable to differentiate between hdb and additional tiers.
1.9.0
Release Date
2024-03-26
New Features
-
Developer Experience Improvements for Packaging
kdb Insights Enterprise packaging now allows clients to manage database and pipeline configuration as well as custom code within a single package. This greatly simplifies developer and operational workflows, as users can maintain and administer databases and pipelines within a single package and with specific configuration for each pipeline. Packaging has also been integrated with the UI so that any databases and pipeline created using the CLI using the
kxi package
command can be viewed and maintained by the UI and vice versa. Refer to Packaging for more details. -
SQL Enhancements
SQL in kdb Insights Enterprise now includes support for ORDER BY and LIMIT clauses, in addition to other functions. Changes to how SQL statements are parsed also significantly improves SQL query performance in this release. Query performance, on average, is up to 3x faster than before. Refer to SQL 2 for more information on the new SQL implementation and its current Known Limitations.
-
Streaming to Views
In the kdb Insights Enterprise UI, Views can now subscribe to real-time data from a pipeline, rather than relying on intermittent polling. This gives you the ability to visualize and analyze your data immediately. Refer to the streaming guided tour for details on configuring pipelines and views for this feature.
Streaming to KX Views is a BETA feature in kdb Insights Enterprise 1.9.0.
· Refer here to the Standard Terms related to Beta features.
· We invite you to use this Beta feature and to provide feedback using the portal.
-
Snapshots & Majority Archival for Reliable Transport
Reliable Transport now supports taking snapshots allowing for majority archival of logs files. Previously, all nodes needed to be healthy before RT logs could be archived.
-
Configurable Connections for Reliable Transport
Reliable Transport now supports subscribers and publishers setting customized parameters when they register with their respective sequencer cluster. This gives you greater configurability in how you deploy and orchestrate components of RT. See rt.qpk for more details.
Improvements
-
System Resources Configuration
Defining system resources has been streamlined, enabling you to set all aspects of resource allocation from a single configuration page in the kdb Insights Enterprise UI. See Adjust Resources for more information on this new tab.
-
Batch Delete
Batch delete now allows you to delete from all tiers of your HDB. Previously, this was only possible for the tier with the most recent HDB data. See Database Delete for details on how to perform a delete.
-
Python Cloud Readers
Python cloud readers now support polling for new matches to a glob pattern with the
watch
parameter. See Readers for details. -
Upgrade to Insights Core 4.1.1
kdb Insights Enterprise images have been upgraded to run Insights Core 4.1.1, which runs kdb+ 4.1 2024.03.12.
-
Upgrade to RockyLinux 9
kdb Insights Enterprise images have been upgraded to run on Rocky Linux version 9 as their base operating system. This addresses security vulnerabilities present in version 8.
-
Stream Processor Troubleshooting Guide
There is now a troubleshooting guide{:target="blank"} available to help users better understand how to debug their pipelines running in _kdb Insights Enterprise.
Fixes
Expand to see the full detail of resolved issues here
Database
-
If EOD failed while in the middle of an HDB transaction, the aborted transaction was not properly cleaned up after the failure and could result in data inconsistency after reconnection.
-
The EOD process was not performing a garbage collection after a batch ingest.
-
Data was unqueryable after a batch ingest if no additional streaming data was loaded after the initial batch ingest.
Reliable Transport
- In deployments with a one-node RT configuration, RT has been updated to roll its log file after a restart. This prevents data flow from being paused. The RT log file will roll from
log.x.y
tolog.x.y+1
.
UI
-
The Subscriber node was limited to only specifying one keyed column. This has been resolved to allow one or more keyed columns, ensuring that streaming supports a record being streamed per unique combination of all specified keyed columns and to allow filtering on any one of the keyed columns.
-
When configuring a Schema node, the regular expression pattern for table validation has been updated to allow single character column names.
-
For Function nodes, users can now customize their function parameters to accept the metadata and operator, in addition to data, in the Pipeline UI. See here for details.
-
The
Error Mode
in the Pipeline Settings section of the Pipeline editor has been removed as it is no longer required, due to a stack trace now being logged when user code errors. -
The logging level within the Runtime Pipeline Settings section of the Pipeline editor was not correctly displaying or setting the log level of the SP worker. This has been resolved.
-
There was no way to specify
\t
as the delimiter in a CSV decoder in the UI. This has been corrected in the CSV Decoder node to allow for any delimiter to be used. -
After a pipeline has been deployed, when the status is either “Running” or “Finished”, a user is able to delete the pipeline. When the pipeline is deleted, the Overview page was still displaying it under “Running Pipelines”. This has been resolved to not display deleted pipelines.
-
Disk sizes for IDB and HDB tiers can now be set to any value greater than 0.
-
Users without the permission to create a pipeline should not be able to use the Import Data button on Database Page. This has been resolved.
-
Deleting running databases is now allowed.
Views
-
Notifications in Views were not reporting correctly. This has been resolved.
-
Under certain circumstances when you had a particular data source selected in the Data Source window and clicked New the existing values remained visible, however they were all ignored. This has been resolved.
-
When using the
getData
API withagg
set tosym;distinct;sym
, the UI was ignoring this request and not sending any aggregation. This has been resolved. -
Switching between tabs caused the user-selected date to switch back to the default setting. This caused the View to constantly reload. This has been resolved.
Stream Processor
-
When using Python, 'sp.decode.csv' did not support reading columns as strings. This has been updated to enhance flexibility when reading table columns. It now accepts an empty string "" in lieu of a schema to read in all table columns as strings.
-
Sidecar containers no longer log at a
TRACE
level by default and now adhere to the log level that is set by the user. -
The SP controller now adheres to the log level set by the user.
-
The CSV decoder did not support decoding csvs which had line-returns (i.e. \n’s) embedded in strings. This functionality can now be turned on with a new optional parameter called Strings Contain Newlines in the UI and
newlines
in the q and Python APIs. -
When running a large import from a static source (file, Azure, AWS, GCP), there was no indication of progress in the logs. The SP logs now displays the file offset periodically while processing to indicate on-going progress of the import.
-
The SP’s status APIs will now return
TEARING_DOWN
rather thanNOTEXIST
when a pipeline has PVCs that are still in a terminating state. -
Under certain circumstances when you added an additional stream to your database it could become orphaned after you renamed the database. This has been resolved.
-
Under certain circumstances when creating a pipeline using the Import feature the Parse Strings column was missing from the Apply Schema page, causing the pipeline to fail when deployed. This has been resolved.
-
The running pipelines section displays a count of warnings and errors that have occurred in the last 10 minutes. Not all of these warnings were being displayed in the Diagnostics page. This has been resolved.
CLI
-
When trying to enroll a client with a topic, the client was enrolled but the specified topic was not being set. This has been resolved.
-
When a user tried to authenticate but did not have a local browser installed on their machine, the redirect URL was
localhost
instead of the value specified. This has been resolved such that users with a local browser installed can still successfully authenticate. -
Installation with a
kc.lic
license using the--license-filepath
command-line parameter was generating an incorrectvalues.yaml
file. This has been resolved so the license values correspond to the license name. -
Specifying a different hostname as part of a
kxi
command is ignored after authentication, as you can only be authenticated to one kdb Insights Enterprise host at a time. For example, if you authenticate to HOSTNAME1 by callingkxi --hostname HOSTNAME1 auth login
you are free to make calls against this host. If you attempt to specify a different hostname in subsequent callskxi --hostname HOSTNAME2
, the--hostname
parameter will be ignored and the command executed against HOSTNAME1. You must re-authenticate against HOSTNAME2 before running commands against it. -
The function
.kxi.udfs.list.all[]
was failing with a length error. This has been resolved. -
Global environment variables set within your
values.yaml
will now be set by all database and assembly deployments.global: env: env: var
Query Environments
- Query environments with entitlements enabled required the creation of separate entitlement for the
-qe
assembly in order for queries to work. This has been resolved.
Upgrades
- Fixed an issue where it was possible for pods to be unscheduled after an upgrade or rollback due to a volume node affinity conflict. Please note that this is still possible when rolling back from 1.9 to earlier versions. Please check the release notes of the target rollback version to address.
Security
- Various CVEs have been remediated as part of this release.
Third-party Dependencies
kdb Insights Enterprise 1.9.0 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.9.0 release of the standalone infrastructure installation scripts.
Deprecation
-
The CLI
kxi assembly
command is being deprecated in 1.9.0 in favor ofkxi package
and will be removed in a future version. See kdb Insights CLI Reference for more information on thekxi assembly
command. -
prtnEndCB
andreloadCB
callback functions are being deprecated in 1.9.0 and may be removed from future releases. They have been replaced bylifecycle.partition.end
andlifecycle.reload.post
respectively. Their usage can be seen below.
.custom.da.prtnEndCB:{[event;data]
log.info("lifecycle.partition.end invoked.")
}
.custom.da.reloadCB:{[event;data]
log.info("lifecycle.reload.post invoked.")
}
.ev.add[`lifecycle.partition.end ;`.custom.da.prtnEndCB];
.ev.add[`lifecycle.reload.post ;`.custom.da.reloadCB];
- RT
maxLogSize
is no longer used in configuration and will be ignored from older configurations. RT will now utilize all disk available to it.
Artifacts
KX Nexus
The KX Nexus repository will be sunsetted in the future. Nexus links on this page are provided as a temporary alternative to the KX Downloads Portal for existing users. The KX Downloads Portal is the preferred repository for all use cases and KX Nexus links will be removed once the KX Nexus repository is decommissioned.
type | Nexus location | Downloads Portal location |
---|---|---|
Enterprise | insights-1.9.0.tgz | insights-1.9.0.tgz |
Operator | kxi-operator-1.9.0.tgz | kxi-operator-1.9.0.tgz |
CLI | kxicli-1.9.0-py3-none-any.whl | kxicli-1.9.0-py3-none-any.whl |
RT C interface | kxi-c-sdk 1.9.0 | kxi-c-sdk 1.9.0 |
RT Python interface | kxi-rtpy-1.9.0 | kxi-rtpy-1.9.0 |
RT Java interface | kxi-java-sdk 1.9.0 | kxi-java-sdk 1.9.0 |
RT Bridge | rt-bridge 1.7.0 | rt-bridge 1.7.0 |
Infrastructure | kxi-terraform-1.9.0.tgz | kxi-terraform-1.9.0.tgz |
Upgrade Notes
-
Keycloak and Postgres Values Changes
The values file settings for the Keycloak and Postgres components have changed for 1.9. The panels below describe the modifications you need to make to this file for all standard or shared Keycloak deployments before upgrading to 1.9.
Merging install values
The values below are a partial extract of your deployment values. These should be merged with your existing values file.
For more information, refer to Applying configuration changes.
A standard deployment deploys Keycloak as part of the application. If you're using this configuration, you should ensure your values file includes the details shown below.
The names
kxi-keycloak
andkxi-postgresql
are the default values, update these as appropriate.global: keycloak: auth: existingSecret: kxi-keycloak postgresql: auth: existingSecret: kxi-postgresql .. keycloak: auth: existingSecret: kxi-keycloak postgresql: auth: existingSecret: kxi-postgresql ..
Keycloak values
Please note there are two sets of
keycloak
andpostgres
objects;global.keycloak
,global.postgresql
,keycloak
,keycloak.postgresql
.Both sets of these are required as shown above.
If deploying using the shared Keycloak mode, configure the values file as described below.
The
KEYCLOAK_RELEASE_NAME
andKEYCLOAK_NAMESPACE
variables should be updated to match those of your Keycloak deployment. The shared Keycloak docs explain each of these values.global: keycloak: authURL: http://${KEYCLOAK_RELEASE_NAME}.${KEYCLOAK_NAMESPACE}.svc.cluster.local/auth/ auth: existingSecret: kxi-keycloak postgresql: auth: existingSecret: kxi-postgresql ..
-
If a user has an existing
kxi-license
secret created from akc.lic
ork4.lic
,kxi install setup
will re-use this secret, but will not set the appropriate values in the resultingvalues.yaml
file. See available license types Setup - kdb products. -
The default image repository used for new 1.9 installations will be the KX Downloads Portal. The credentials to access this are stored in a docker config json secret named
kxi-image-pull-secret
. If you are upgrading from an earlier version that used the KX Nexus image repository with an exiting secretkxi-nexus-pull-secret
, this existing repository and secret will be used. -
After executing an upgrade, if an RT pod does not appear as ‘Ready’, the pod should be deleted/killed so that it is restarted.
-
During an upgrade, the RT pods attempt to delete a Kubernetes configmap. Since there were 3 RT pods, only one of the pods will be successful in deleting the config map. This means that the other pods will receive ERRORs back indicating that their delete attempt failed. These
Failed to delete snapshot: tmp
errors are benign and will be set to WARN messages in future. -
Query Labels Style
KXI_ALLOW_OLD_LABEL_STYLE
now defaults to "false". It must be explicitly set to "true" in the RC to allow the "old label style". For example, in a system with labelsregion
, andclass
:-
Labels in queries (such as
.kxi.getData
) must be defined in thelabels
parameter.- old style:
--data '{"table": "trade", "region": "usa", "class": "equities"}'
- new style (new default):
--data {"table": "trade", "labels": {"region": "usa", "class": "equities"} }'
- old style:
-
Labels in SQL queries must be prefixed with
label_
.- old style:
SELECT * FROM trade WHERE region='usa', class='equities'
- new style (new default):
SELECT * FROM trade WHERE label_region='usa', label_class='equities'
- old style:
-
Rollback
-
If performing a rollback from 1.9.0 to any earlier version, assemblies must be running prior to initiating the rollback to ensure no data is lost. See Upgrading for how to perform upgrades/rollbacks and the available commands to use. Please see the following specific instructions per version:
Rollback from 1.9 to a version less than 1.7 is not possible.
For rollback from 1.9 to a prior version:
- Confirm all assemblies are running before initiating rollback.
- Execute the rollback operation. Note that the
kxi-rt-rollback
andkxi-sp-rollback
charts are run automatically as part of the rollback operation as of 1.9.0.
For rollback from 1.8, the following instructions should be followed:
- Teardown your assemblies.
- Run the
kxi-sp-rollback
chart. If the target version for rollback is less than 1.7, also run thekxi-db-rollback
chart. - Rollback kdb Insights Enterprise per the rollback instructions referenced above.
- Redeploy assemblies.
Known Issues
Expand to see the list of known issues here
Upgrades & Rollbacks
-
Users are prompted to tear down assemblies upon rollback. The rollback should exit if the user responds ‘n' to this prompt. Instead, the rollback continues.
-
Occasionally, rollbacks may fail with an error when running
kxi install rollback
. This is due to timeouts with the Helm rollback, resulting in errors such as the following:timed out waiting for the condition context deadline exceeded
To resolve this issue, re-run the rollback.
Pipelines
-
It is currently possible to rename a pipeline in the UI to a pipeline name that is already in use. Until this is resolved, ensure your pipeline names are unique.
-
When deploying a pipeline that writes to a table that does not exist, no error will be thrown.
Packaging
kxi.package.load_file
does not work properly for Python files. For a Python module, useimport
instead.
Query Tab
- There is presently no way to select a subset of the available columns during the
getdata
phase in the Query tab of the UI. Conversely, thegetData
API allows this in theagg
parameter where you can pass in the list of columns you want with no aggregation function.
UI
-
Changing the system log level in the UI does not get actioned.
-
On Mozilla Firefox, when clicking
Deploy
in aDatabase
screen, the scroll wheels do not render properly in the pop-up.
Views
-
When displaying streaming data in Views and using the Map component, the map may take some time before the streaming data is displayed. A page reload may be required.
-
When working with a View then temporarily navigating away to a different tab, any changes made will be saved despite the user not requesting this. If multiple users are editing the same view, any time one switches away from the tab (even if they haven’t made any changes), the current version of their view will be saved, which may not include changes made other users. To resolve this, use the Undo action to return the last desired saved state.
-
A popup may be shown indicating there may be unsaved changes when no changes have been made.
Scratchpads
-
Certain Python expressions cause internal errors in Python scratchpads:
lambda x: x * 2 memoryview(bytearray("myTest", encoding='utf-8')) {1,2,3}
Reliable Transport
- RT clients output certain logs to a folder determined by the
KXI_C_SDK_APP_LOG_PATH
environment variable ($KXI_C_SDK_APP_LOG_PATH/kxi_c_sdk_logs/rt_helper.VMC7AW/rt_helper.log
). This variable uses a default location of/tmp
. When there are recurring errors, these logs can fill the/tmp
directory which is only 5MB in size. To overcome this issue, you can change this environment variable. See Environment variables for steps on how to change this.
Deprecation
-
The CLI
kxi assembly
command is being deprecated in 1.9.0 in favor ofkxi package
and may be removed from future releases. See kdb Insights CLI Reference for more information on thekxi assembly
command. -
prtnEndCB
andreloadCB
callback functions are being deprecated in 1.9.0 and may be removed from future releases. -
RT
maxLogSize
is no longer used in configuration and will be ignored from older configurations. RT will now utilize all disk available to it.
1.8.2
Release Date
2024-01-23
Fixes
Expand to see the full detail of resolved issues here
Database
- An issue has been resolved in relation to Batch Ingest. Previously, some data in the Historical Database (HDB) was persisted to disk but it was unqueryable. This has been fixed, and all data ingested into the HDB is now queryable.
Third-party Dependencies
kdb Insights Enterprise 1.8.2 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.8.2 release of the standalone infrastructure installation scripts.
Artifacts
Type | Location |
---|---|
Enterprise | insights-1.8.2.tgz |
Operator | kxi-operator-1.8.1.tgz |
CLI | kxicli-1.8.0-py3-none-any.whl |
C interface | kxi-c-sdk 1.8.0 |
RT Python interface | kxi-rtpy-1.8.0 |
Java interface | kxi-java-sdk 1.6.0 |
RT Bridge | rt-bridge 1.7.0 |
Infrastructure | kxi-terraform-1.8.0.tgz |
1.8.1
Release Date
2024-01-16
Improvements
-
Encryption of data in transit with OpenShift
The encryption of data in transit is now supported when deploying kdb Insights Enterprise using OpenShift.
-
Stream Processor CSV Decoder Improvements
The Python CSV decoder now supports the
first
header option. This should be used when only the first batch of your dataset contains a header in the CSV file. See Decoders for information on the CSV decoder configuration.
Fixes
Expand to see the full detail of resolved issues here
Packaging
- Resolved an issue where
.kxignore
files were incorrectly read as bytes instead of strings when compressed on Ubuntu for Windows.
Stream Processor
-
Stream Processor Beta features configuration has been fixed. When deploying the SP Helm chart, setting
betaFeatures: true
wasn't correctly enabling Beta features. -
A fix has been introduced in the UI to ensure changes to the position of an SP node in a pipeline are now correctly saved.
-
When creating a pipeline in the UI, the Stream Processor S3 writer now allows for proper writing to an object storage bucket. Previously, this could error with
Some nodes have errors, fix them before trying to deploy this pipeline
. -
Previously, the CSV decoder could drop messages when paired with a file reader (Amazon S3, Google Storage, Azure Storage) where chunking was enabled and the CSV decoder header was set to
always
. This issue has been resolved. See Decoders for information on the CSV decoder configuration. -
SP Cloud Readers no longer error when there are no files matching the expected pattern. Previously, a cloud reader using glob pattern matching would prevent the pipeline from starting if no files matched. In particular, this would occur when trying to use the file watcher feature and no files matched as of the pipeline starting.
-
The delimiter select field in the Stream Processor CSV node has been updated to allow any delimiter to be used. Previously, there was a restriction on the number of characters that could be submitted, preventing entry of
\t
or any other delimiter. -
Resolved the problem of all window nodes triggering their jobs at the same time, despite their configuration settings. This would occur in setups with pipelines with multiple window nodes, or separate pipelines that read and write to the same RT stream. See Windows nodes for more details.
Security
- Various CVEs were remediated as part of this release.
Third-party Dependencies
kdb Insights Enterprise 1.8.1 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.8.1 release of the standalone infrastructure installation scripts.
Artifacts
type | location |
---|---|
Enterprise | insights-1.8.1.tgz |
Operator | kxi-operator-1.8.1.tgz |
CLI | kxicli-1.8.0-py3-none-any.whl |
C interface | kxi-c-sdk 1.8.0 |
RT Python interface | kxi-rtpy-1.8.0 |
Java interface | kxi-java-sdk 1.6.0 |
RT Bridge | rt-bridge 1.7.0 |
Infrastructure | kxi-terraform-1.8.0.tgz |
1.8.0
Release Date
2023-12-11
New Features
-
Data Encryption
Data in transit within kdb Insights Enterprise can now be encrypted, providing enhanced security. This feature meets compliance and regulatory requirements and protects against cyber-security attacks. For further details on how to enable encryption via the CLI, see Encryption of data in transit.
-
OpenShift Container Platform (OCP) Support
kdb Insights Enterprise can now be deployed on premise using the OpenShift Container Platform (OCP). This expands on the deployment topology options of kdb Insights Enterprise, and enables customers to take advantage of its capabilities without having to run in a public cloud. For further details, see OpenShift infrastructure prerequisites.
Note
See here for details on an issue with Openshift upgrades.
-
Azure and GCP Parquet Reader Support
The kdb Insights Parquet Reader functionality has been extended to support Azure and GCP object storage. Users can now read, decode, and watch Parquet files stored on Azure and GCP, as well as AWS. These enhancements are available in the UI, the q API and the Python API.
-
HTTP Publishing
kdb Insights Enterprise has introduced a REST interface to publish data over HTTP. This expands the interoperability of kdb Insights Enterprise for ingesting data. Note that file size is limited to 10MB per publish. See Upload via HTTP for more details.
-
Python Interface to Reliable Transport
A new Python interface allows you to publish data to Reliable Transport from your own Python application in Docker, or on Linux or Windows. See here for details.
-
New Database Preview API
You can now query a sample of your data from the database to understand your data/schema, construct more complex queries, or build visuals on the full data set. See here for details.
Improvements
-
Autoscaling Service Gateways
The Service Gateway and Aggregator can now autoscale if CPU exceeds your configured threshold. The default is to trigger scaling when CPU exceeds 80% for 5 seconds, with a default stabilization period of 300 seconds. All of this is configurable in your values file. See here for details.
-
CLI Enhancements
The kdb Insights Enterprise CLI contains a number of security and user experience improvements added:
- Browser-based authentication for users, and the ability to authenticate even without a browser
- Ability to authenticate once and still straddle administrator and developer views
- Token caching & refreshing to support the authentication methods
- Use of profiles to allow for concurrent sessions in different environments
See CLI Authentication for details.
-
UI Enhancements
-
When editing a pipeline node, you are now reminded to apply your current changes before moving away to perform other operations. This avoids the potential loss of changes when clicking on to another node.
-
Ability to view the rolled-up status of a pipeline group when there are multiple pipelines/replicas.
-
Undo and Redo buttons have been added to the Pipeline editor.
-
Pipeline settings have been moved into a new ‘Settings’ tab on the Pipeline page. See here for details.
-
For tables marked in a foreign key relationship, further validation has been added upon startup to check that these exist and fail fast if any are missing.
-
The autocomplete widget in code editors now only appears when manually triggered with Ctrl + Space.
-
-
Stream Processor
-
Cloud file readers in the Stream Processor now support watching for new files. You can now provide a pattern expression to search for file names. When watching is enabled, the file reader will poll for new files that match the expression. For full details on file watching, see the Amazon S3 reader, the Azure Blob storage reader and the Google Cloud storage reader.
-
You can now deploy replicas of pipelines. When deploying, select the “Advanced” option to configure a “Replicas” input field. The environment will deploy the configured number of replicated pipelines identical in configuration with a different ID. See here for details.
-
Fixes
Expand to see the full detail of resolved issues here
Database
-
Improved startup performance of HDB DAPs when loading late data from IDB tier mount.
-
Storage Manager resiliency improvements have been made to improve how SM cleans up aborted transactions after a failure event.
-
Improved validation for schemas in the UI. Rules now validate against invalid configuration combinations with attributes, sorts and reserved words.
-
Improved failure recovery for object storage in Storage Manager. This resolves a case where EOI would not complete and would report
SM Flushing filesystem with [directory]
errors.
Reliable Transport
- Logging for LEADER/FOLLOWER updates have been changed to DEBUG level to reduce log noise.
Stream Processor
-
Previously, it was possible that
FINISHED
SP pipelines using multiple partitioned workers could spawn extra zombie workers. This has been resolved. -
Pipeline quick tests no longer fail if they contained a kdb Insights Stream or kdb Insights Database reader or writer.
-
Previously it was not possible to maintain state of any object that contained a kdb guid type. This is now fixed. However, if a rollback to an earlier version is required after an upgrade, a rollback step should be run so that on-disk checkpoints can be converted to a format compatible with earlier versions.
-
Fixed an issue where the csv decoder, with option
header=first
, and paired with an unbounded streaming reader, was incorrectly dropping the first record in every stream batch. -
Fixed a determinism gap when reading objects from cloud storage using a glob pattern. In a recovery scenario, it was possible to receive duplicate data.
-
Fixed a bug where unpopulated column of type 0h (i.e. list) was not getting filled in with blank entries correctly.
-
If a pipeline is deployed as part of an assembly,
.qsp.write.toDatabase
now defaults to writing to that assembly. This means the assembly name can be omitted when calling.qsp.write.toDatabase
, for example,.qsp.write.toDatabase[
trade; .qsp.use`directWrite!(::;1b)]
. -
Fixed a bug where in a single path pipeline, the existence of more than one emitter resulted in that same number of combine nodes being added.
Upgrades
- Resolved a previously-published known issue where upgrades could occasionally fail with an error when running
kxi install upgrade
. This occurred due to a race condition in the 1.7.0 version of the CLI when trying to replace the CRDs.
File Watchers
-
When using the file/cloud readers in a non-file watching scenario, files that were not getting chunked did not have their state checkpointed. This meant that during a recovery scenario, duplicate data would be generated for those files.
-
An issue has been resolved that could cause the file watcher to lose data at the boundaries when chunking was set to
auto
.
Security
- Various CVEs were remediated as part of this release.
User Interface
-
Resolved the issue where the resizing of a browser window threw the error
Error: Cannot read properties of undefined (reading 'getBoundingClientRect') or this.getElements()[0] is undefined
. -
Resolved an issue where certain Python libraries such as
spacy
,tensorflow
,keras
did not work in the Scratchpad. -
Postgres and SqlServer nodes no longer incorrectly have a password field.
-
Resolved the issue where the Views page in the UI occasionally froze when performing certain operations using the map.
-
The instances where the error "Failed to compare saved assembly [assembly name] to deployed version" were erroneously triggered are now resolved.
-
Resolved the issue where resizing your browser window throws "Error: Cannot read properties of undefined (reading 'getBoundingClientRect') or this.getElements()[0] is undefined".
-
Duplicate filters are no longer being reported after creating then saving a filter.
-
Navigating to a Dashboard/Screen from Actions is now working in Views. Previously, you could receive the error "Error: Cannot read properties of undefined (reading 'navigate')" or be redirected to the homepage.
-
Creating a View with a Datagrid, copy it to a second View and pasting, and then returning to your original View would cause your changes to be lost. This has now been resolved.
-
Fixed the incorrect JSON file format that was causing issues when trying to download pipeline logs from the UI.
Third-party Dependencies
kdb Insights Enterprise 1.8.0 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.8.0 release of the standalone infrastructure installation scripts.
Artifacts
type | location |
---|---|
Enterprise | insights-1.8.0.tgz |
Operator | kxi-operator-1.8.0.tgz |
CLI | kxicli-1.8.0-py3-none-any.whl |
C interface | kxi-c-sdk 1.8.0 |
RT Python interface | kxi-rtpy-1.8.0 |
Java interface | kxi-java-sdk 1.6.0 |
RT Bridge | rt-bridge 1.7.0 |
Infrastructure | kxi-terraform-1.8.0.tgz |
Upgrade notes
-
Encryption of data in transit{:target="blank"} is turned on by default for fresh installs. When you upgrade _kdb Insights Enterprise the CLI gives you the option to turn encryption on or off. To change this setting after installation, perform an upgrade.
-
When creating a new database through the kdb Insights Enterprise UI, the
isSharded
schema property is now being defaulted to true. See here for more details on the implications of partitioned tables.
Known Issues
-
Upgrading kdb Insights Enterprise on Openshift will report that the
insights-on-k8s
task will fail to install Istio. This can be safely ignored. Please follow the instructions to install Istio as a prerequisite. -
Using a Stream Processor Amazon S3 reader may fail when running on EKS. To workaround this, please set the
tenant
field tokxinsights
. -
If you have has an existing
kxi-license
secret created from akc.lic
ork4.lic
,kxi install setup
will re-use this secret, but will not set the appropriate values in the resultingvalues.yaml
. See the available license types here. -
Occasionally installs and upgrades may fail with an error when running
kxi install
. This can occur due to timeouts with the helm upgrade, giving errors such as the below:timed out waiting for the condition context deadline exceeded
Re-run the install or upgrade to resolve this issue.
-
With entitlements enabled, the Query tab in the UI will not return data as there are issues with resolving the correct database name when checking entitlements. In order to workaround this issue, you should create a second entitlement for each database / assembly.
For example if you create and entitle an assembly called
dfx-assembly
, you should also create an identical entitlement calleddfx-assembly-qe
. This-qe
entitlement should be kept in-sync when updating the main one.This restriction will be removed in a future release.
-
qsql
requests, made either through theQ
tab (Query environment) in the UI or via REST to/qsql
, do not work with entitlements. The.kxi.qsql
API should be disabled if using entitlements. This is disabled by default in kdb Insights Enterprise. See here for more details.
1.7.4
Release Date
2024-01-04
Fixes
Expand to see the full detail of resolved issues here
Database
-
Improved failure recovery for object storage in Storage Manager. This resolves a case where EOI would not complete and would report
SM Flushing filesystem with [directory]
errors. -
Resolved a bug in which EOIs were not completing, and
NoSuchKey
errors were occurring after moving data to object storage. -
Improvements have been made to the refresh token logic for AWS deployments. Previously, token refresh logic was erroneous if the session was not requesting object storage bucket data. The refresh rotated every 30 minutes instead of at the session expiry time. Session expiry time is configured with EC2 instance metadata, or by
KX_KURL_AWS_TOKEN_DURATION
when using Amazon STS.
User Interface
-
Resolved an issue where certain Python libraries such as
spacy
,tensorflow
,keras
did not work in the Scratchpad. -
In the Query tab, duplicate filters are no longer being reported after creating then saving a filter.
Security
- Various CVEs have been remediated as part of this release.
Third-party dependencies
kdb Insights Enterprise 1.7.4 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.7.4 release of the standalone infrastructure installation scripts.
Artifacts
type | location |
---|---|
Enterprise | insights-1.7.4.tgz |
Operator | kxi-operator-1.7.2.tgz |
CLI | kxicli-1.7.2-py3-none-any.whl |
C interface | kxi-c-sdk 1.7.0 |
Java interface | kxi-java-sdk 1.6.0 |
RT Bridge | rt-bridge 1.7.0 |
Infrastructure | kxi-terraform-1.7.0.tgz |
1.7.3
Release Date
2023-11-01
Fixes
Expand to see the full detail of resolved issues here
Database
- Optimizations made to the
getData
API to improve query performance when theprtnCol
is sorted.
Security
- Various CVEs have been remediated as part of this release.
Third-party dependencies
kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.
Artifacts
type | location |
---|---|
Enterprise | insights-1.7.3.tgz |
Operator | kxi-operator-1.7.2.tgz |
CLI | kxicli-1.7.2-py3-none-any.whl |
C interface | kxi-c-sdk 1.7.0 |
Java interface | kxi-java-sdk 1.6.0 |
RT Bridge | rt-bridge 1.7.0 |
Infrastructure | kxi-terraform-1.7.0.tgz |
1.7.2
Release Date
2023-10-03
Improvements
-
SQL Queries
- SQL queries will now use less memory if late data is enabled and you are querying data for the current day.
Fixes
Expand to see the full detail of resolved issues here
Database
-
Resolved an issue where data for dates modified after the retention period was not getting moved from the first to the secondary HDB tier.
-
Improved the startup time for HDB DAPs, allowing for faster availability of processes that serve queries.
Stream Processor
- The Apply Schema transform node now works with NULLs in string types.
Third-party dependencies
kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.
Artifacts
type | location |
---|---|
Enterprise | insights-1.7.2.tgz |
Operator | kxi-operator-1.7.0.tgz |
CLI | kxicli-1.7.1-py3-none-any.whl |
C interface | kxi-c-sdk 1.7.0 |
Java interface | kxi-java-sdk 1.6.0 |
RT Bridge | rt-bridge 1.7.0 |
Infrastructure | kxi-terraform-1.7.0.tgz |
1.7.1
Release Date
2023-09-19
Fixes
Expand to see the full detail of resolved issues here
Database
-
Resolved an issue with the Storage Manager not rolling back aborted transactions if an EOD failed.
-
Deleting data from the HDB using a
filter
option now correctly performs the delete. Previously, this returned an error claiming that the column(s) being filtered didn’t exist. -
EOIs are no longer delayed after an EOI recovery event. Previously, it was possible for EOIs to become held up if the process was terminated then subsequently recovered. The EOIs would still complete successfully, but with a delay.
Security
- Resolved CVE-2023-37920 Certifi's removal of e-Tugra root certificate, by upgrading the spaCy dependency.
Stream Processor
- Attempts to horizontally scale pipelines using the direct write feature will now explicitly error since this is not currently supported.
Deployments & Upgrades
- Resolved the known issue in 1.7.0 in which a race condition could cause an upgrade to fail with an error when running
kxi install upgrade
.
Third-party dependencies
kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.
Artifacts
type | location |
---|---|
Enterprise | insights-1.7.1.tgz |
Operator | kxi-operator-1.7.0.tgz |
CLI | kxicli-1.7.1-py3-none-any.whl |
C interface | kxi-c-sdk 1.7.0 |
Java interface | kxi-java-sdk 1.6.0 |
RT Bridge | rt-bridge 1.7.0 |
Infrastructure | kxi-terraform-1.7.0.tgz |
1.7.0
Release Date
2023-08-25
New Features
-
End-of-day writedown memory controls
The Storage Manager (SM) process now supports restricting memory consumption for data writedown, giving the opportunity for cost savings. Reducing the RAM allocated to SM may cause EOD to run longer than if RAM was unrestricted.
For guidance on tuning your RAM setup, see Performance best practices and the recommended system sizes for Azure here.
-
One-node RT
kdb Insights Enterprise now allows you to optionally deploy an RT Stream as a one-node cluster instead of a three-node cluster when deploying via the kdb Insights CLI.
- In a three-node cluster, RT provides fault tolerance and high availability (HA).
- In a one-node cluster, RT provides fault tolerance, but the cluster is not highly available. This configuration saves resources and reduces costs for workload use cases that do not need to be highly available. In the event that the single RT node goes down, the publisher then continues to write to its local RT log file. When the RT node restarts, the publisher retrieves any data that is present on the publisher node that it has not yet received.
See here for more details on RT architecture.
-
C Subscriber support in Reliable Transport
The C language interface for RT is now able to subscribe to the data on an RT stream. See here for more details.
-
Backfill a new column with derived data
When adding a new column to your schema, you can now define a custom function that will backfill the new column with derived data.
-
Publishing derived data from custom APIs
Custom APIs that generate derived data can now stream and/or persist their output.
-
Appending data on batch ingest
When ingesting data using the direct write feature, you now have the option to append data rather than replace it. See overwrite option for more details.
-
Data Entitlements for querying data Beta
When querying via REST or IPC, kdb Insights Enterprise now provides data entitlement restrictions for limiting user access to the data contained within specific deployed databases.
-
Parquet file reader Beta
Ingesting Parquet files from local storage or from an AWS S3 bucket using the q reader API, Python reader API and the UI is now enabled. This feature will be enabled for Azure and GCP and include a file watcher with file pattern matching in a future release.
Improvements
-
End-of-day writedown performance improvements
The Storage Manager (SM) process has been optimized to significantly improve its performance - up to 50-70% reduction in EOD duration. This improved EOD performance can reduce the memory allocated or the time taken for writedown, depending on your workloads and configuration.
For guidance and best practices on how to configure your schema and system to take advantage of these optimizations, review Performance best practices and the recommended system sizes for Azure here.
-
User interface
-
The database schema editor has been completely redesigned with a new responsive table grid for entering column metadata details. The editor uses keyboard driven actions for efficient data entry. Backed by HTML canvas rendering, hundreds of columns can be added with ease.
-
The Database configuration now supports different storage classes for the IDB and HDB (for example, FSx for Lustre). There is now a dropdown under Advanced Writedown settings for the
File System
that changes the mount storage class. See here for details. -
The code editors in the Pipeline Editor can now be used to evaluate q and Python code. See Executable Editors in the
Building a Pipeline
section for more information. -
Virtual queries are now available in Views. See Virtual data sources for more information.
-
The ‘Database Wizard’ has been removed from the UI. Databases can now be created and configured using the unified database configuration screens.
-
Query results in the UI are now shown using the timezone as returned by the query APIs for both Console and Table views. Previously, Table view would apply the user’s timezone which may have been different.
-
-
Readers
-
When using cloud and file readers, you can now use filename pattern matching to ingest multiple files using a single reader. This expands the ability for you to build up ingest workloads that pull in multiple files. For example, using an S3 reader, you could read all CSV files in all folders within a bucket using the pattern:
s3://path/to/bucket/**/*.csv
-
When using cloud and file readers you can now use a file watcher which is usually paired with filename pattern matching described above. The reader will continually watch for new files matching the file path(s) provided. See here for details on:
- q APIs: .qsp.read.fromGoogleStorage, .qsp.read.fromAzureStorage and .qsp.read.fromAmazonS3.
- Python APIs: .qsp.read.fromGoogleStorage, from_azure_storage and from_amazon_s3.
- UI: Google Cloud Storage, Microsoft Azure Storage and Amazon S3.
Not to be used with the
Write Direct to HDB
optionThe Watcher options should not be used when also choosing the
Write Direct to HDB
option in the Database Writer in the UI, the q or Python APIs. -
Advanced configuration options for the Kafka Reader node in the Pipeline Builder UI allow more flexible options for security protocol configuration or changes to fetch intervals, as seen when subscribing to an Azure Event Hub Kafka connector.
-
-
Reliable Transport
- Users can now launch KXI feeds from outside of their Kubernetes cluster and send data to their kdb Insights Enterprise deploy.
-
Logging
-
RT Stream logs are now included in Database logs on the Diagnostics page, and they are also included in the Database Issues counts shown on the Database Overview page.
-
When viewing a deployed pipeline, you can now click on an icon to jump straight to the pipeline logs.
-
-
Terraform Scripts
- The default node types for each cloud provider have been changed to ensure that the default node group uses memory-optimised instances, and the node group for rook-ceph has storage-optimised instances:
AWS: default node pool: "r5.xlarge" rook-ceph node pool: "i3.2xlarge" Azure: default node pool: "Standard_E4_v3" rook-ceph node pool:”Standard_L8s_v2" GCP: default node pool: "n1-highmem-4" rook-ceph node pool:”n2-standard-8"
-
The deployment of the nginx controller has been changed to be a Daemonset which will ensure consistent availability across all nodes.
-
The standalone installation using the terraform client scripts now enables cloud native system metrics on GKE clusters.
Fixes
Expand to see the full detail of resolved issues here
Database
- When using the Filter By Label
option in the Query page, you can now use more than one label for the filter. In previous releases, additional labels beyond the first were not taken into account when the query was executed.
-
Fixed an issue where schema type conversions from strings to GUIDs would fail and cause a type error.
-
Fixed an issue on certain NFS-based system (namely Lustre) where the ‘current’ database pointer could get converted to a directory resulting in an unexpected error.
EISDIR: /data/idb/current.new -> /data/idb/current
-
An error has been fixed that would occur when updating an object storage tier that used an inventory file after a data migration.
-
Fixed an issue when querying with
.kxi.getData
using a groupBy operation on a foreign key column where the name of the foreign key column would be thrown and the query would fail to complete. -
Fixed an issue where renaming columns in the schema UI would incorrectly delete the underlying column and create a new one with null data. Column renames in the UI now correctly preserve the column’s data.
-
A new warning has been added to the Resource Coordinator if no Aggregators connect to it. After a wait period, if an Aggregator continues not to connect, this will be escalated to an error to indicate that the system is not ready for queries.
-
Stream resources now correctly show the amount of resources to be used when an additional stream was added. Previously the stream resources were only showing the compute settings for a single stream instead of adding both together.
-
When the kdb Insights Database was configured with an object storage tier but an inventory file was not provided, the Storage Manager would error on startup with invalid object storage URI error. This has been changed to not attempt to create an inventory file if no configuration is explicitly provided.
-
Resolved an issue when attempting to migrate HDB data to Azure Blob Storage. Previously, this would fail with a signature error.
-
Fixed a potential race condition between NFS replication and DAP reload signals. It was previously possible that a DAP may reload before NFS replicated data which could result in data gaps. A new sequencing mechanism has been added to ensure DAPs do not reload until replication is complete.
-
Values in a symbol column will now be properly available immediately after a batch ingest. Previously, these would be missing until the next EOI.
-
Renaming a sort column now longer errors.
Stream Processor
-
q code in a pipeline’s Global Code field will now be executed with the same semantics as loading a .q file. This fixes issues around expressions incorrectly being concatenated, the handling of comments, and expressions starting with .
-
The return of UDFs when generating q objects is no longer incorrectly wrapped as foreign objects. A user no longer needs to use a null python function following the UDF to unwrap it to the correct type.
Command Line Interface - Reinstalls and rollbacks no longer fail due to password history issues with the initUser.
Third-party dependencies
kdb Insights Enterprise 1.7.0 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.7.0 release of the standalone infrastructure installation scripts.
Artifacts
type | location |
---|---|
Enterprise | insights-1.7.0.tgz |
Operator | kxi-operator-1.7.0.tgz |
CLI | kxicli-1.7.0-py3-none-any.whl |
C interface | kxi-c-sdk 1.7.0 |
Java interface | kxi-java-sdk 1.6.0 |
RT Bridge | rt-bridge 1.7.0 |
Infrastructure | kxi-terraform-1.7.0.tgz |
Upgrade notes
-
Occasionally upgrades may fail with an error when running
kxi install upgrade
. This occurs due to a race condition in the 1.7.0 of the CLI when trying to replace the CRDs. The upgrade will stall and timeout at this point... Replacing CRD assemblies.insights.kx.com Replacing CRD assemblyresources.insights.kx.com Error: Timed out waiting for CRD assemblyresources.insights.kx.com to be deleted
Re-running the upgrade should workaround the issue. This will be fixed in the next patch release.
Known Issues
-
With entitlements enabled, the Query tab in the UI will not return data as there are issues with resolving the correct database name when checking entitlements. In order to workaround this issue, you should create a second entitlement for each database / assembly.
For example if you create and entitle an assembly called
dfx-assembly
, you should also create an identical entitlement calleddfx-assembly-qe
. This-qe
entitlement should be kept in-sync when updating the main one.This restriction will be removed in a future release.
-
Previously cloud readers in a failure recovery scenario would begin reading file(s) from the start, even if they had read them prior to crashing. This would have caused duplicate data if using a stream or database writer. As of 1.7 this is no longer the case. One exception case remains, when using a glob pattern match in a non watching scenario. If the pipeline fails, recovers and matches a new set of files (because the underlying files in the object storage have changed) we may get duplicate data.
-
qsql
requests, made either through theQ
tab (query environment) in the UI or via REST to/qsql
, do not work with entitlements. The.kxi.qsql
API should be disabled if using entitlements. -
Occasionally installs and upgrades may fail with an error when running
kxi install
. This can occur due to timeouts with the helm upgrade, giving errors such as the below:timed out waiting for the condition context deadline exceeded
Re-run the install or upgrade to resolve this issue.
Backward Compatibility
If you are upgrading from a version earlier than 1.6.3 you must also read each of the relevant release notes to ensure you know all the changes that will affect the upgrade. Refer to the previous release page for older releases.
1.6.4
Release Date
2023-11-28
kdb Insights Core Version
This version of kdb Insights Enterprise runs kdb Insights Core 4.0.5.
Fixes
Expand to see the full detail of resolved issues here
Database
-
Improved failure recovery for object storage in Storage Manager. This resolves a case where EOI would not complete and would report
SM Flushing filesystem with [directory]
errors. -
Resolved a bug in which EOIs were not completing, and
NoSuchKey
errors were occurring after moving data to object storage. -
Improvements have been made to the refresh token logic for AWS deployments. Previously, token refresh logic was erroneous if the session was not requesting object storage bucket data. The refresh rotated every 30 minutes instead of at the session expiry time. Session expiry time is configured with EC2 instance metadata, or by
KX_KURL_AWS_TOKEN_DURATION
when using Amazon STS. -
Added a fix for the GCP system command expiry timer being discarded/ignored, which resulted in tokens not refreshing at all using
gcloud auth
.
Security
- Various CVEs were remediated as part of this release.
Artifacts
type | location |
---|---|
Enterprise | insights-1.6.4.tgz |
Operator | kxi-operator-1.6.1.tgz |
CLI | kxicli-1.6.1-py3-none-any.whl |
C SDK | kxi-c-sdk 1.6.0 |
Java SDK | kxi-java-sdk 1.6.0 |
RT Bridge | rt-bridge 1.6.0 |
Infrastructure | kxi-terraform-1.6.1.tgz |
1.6.3
Release Date
2023-07-25
Improvements
- A new quick test deploy option is available for pipelines for faster iteration during development. This deploys a pipeline in test mode to the user’s scratchpad process and is significantly faster than a full test. See quick test deploy for more information.
Fixes
Expand to see the full detail of resolved issues here
Database
-
Fixed a potential race condition between NFS replication and DAP reload signals. It was previously possible that a DAP may reload before NFS replicated data, which could result in data gaps. A new sequencing mechanism has been added to ensure DAPs do not reload until replication is complete.
-
Fixed an issue where converting a column from a
string
to asymbol
would throw atype
error. -
An error has been fixed that would occur when updating an object storage tier that used an inventory file after a data migration.
-
A new warning has been added to the Resource Coordinator if no Aggregators connect to it. After a wait period, if an Aggregator continues not to connect, this will be escalated to an error to indicate that the system is not ready for queries.
-
REST queries could have input parameters converted to the wrong type when multiple type options were possible. REST APIs were converting to the maximum possible type instead of using the correct default type.
-
Database names in diagnostics now correctly correspond to the name of the assembly that the database maps to.
User Interface
-
Any change made on the Database UI tab will now mark the database as unsaved, and warn the user if trying to close the tab with any unsaved changes.
-
The
View Errors
button is now renamed toView Database Errors
and points to the database error logs. To viewPipeline Errors
, click on theOpen Pipeline Errors
icon next to each pipeline error. -
Fixed an issue where renaming columns in the schema UI would incorrectly delete the underlying column and create a new one with null data. Column renames in the UI now correctly preserve the column’s data.
Artifacts
type | location |
---|---|
Enterprise | insights-1.6.3.tgz |
Operator | kxi-operator-1.6.1.tgz |
CLI | kxicli-1.6.1-py3-none-any.whl |
C SDK | kxi-c-sdk 1.6.0 |
Java SDK | kxi-java-sdk 1.6.0 |
RT Bridge | rt-bridge 1.6.0 |
Infrastructure | kxi-terraform-1.6.1.tgz |
1.6.2
Release Date
2023-07-03
New Features
-
A file watcher for polling cloud storage is now supported in the Stream Processor. The file watcher can monitor any Azure, AWS, or GCP object storage directory. It polls for new files to ingest into an SP pipeline. See the q reader API for details on how to configure the
watch
parameter in your SP reader. This feature will be available in the Python API and UI in a future release. -
Query resiliency has been improved by adding the ability to control how processes serving queries are shut down. These processes can now be configured to stay up to serve in-progress query requests before shutting down, ensuring users still receive their data and are unimpacted by system operations. Users simply define how long a process should continue to serve requests before ultimately going offline. This is configured by setting
.values.service-gateway.aggregator.aggregator.terminationGracePeriodSeconds
to an appropriate duration in your values file during installation. -
Query scalability has increased with the new ability to independently scale query processes within an assembly, allowing each assembly to have a tailored amount of these processes. Making this configurable and manually scalable lets you better size your assemblies with specific query workloads in mind. See routing configuration for more details. Furthermore, a new routing parameter called scope supports routing query requests based on your target assembly(s).
-
Reading directly from a Parquet file(s) in local storage is now supported in Stream Processor pipelines, allowing users to take advantage of this efficient common file format. See the q reader API for more details. This feature will be available in the Python API and UI in a future release.
-
Connect your existing kdb database already in object storage to your kdb Insights Enterprise system and query it.
-
Pipelines created in the UI can now be exported to be run by the CLI, either for automation or for migrating between environments. From the UI, select the pipeline and click ‘Export’. See Exporting artifacts for more details.
Fixes
Expand to see the full detail of resolved issues here
User Interface
-
In the kdb Insights Enterprise UI, new validation has been added for table names within schemas to ensure that they do not contain invalid characters. Previously, invalid characters could be included in tables names and the database would error after it was deployed.
-
In the kdb Insights Enterprise UI, you can now allow an internal stream to read data from an external source directly to a database. See stream configuration for more details.
-
When creating a new database or renaming an existing database, you are prevented from entering a database name longer than 24 characters.
Artifacts
type | location |
---|---|
Enterprise | insights-1.6.2.tgz |
Operator | kxi-operator-1.6.1.tgz |
CLI | kxicli-1.6.1-py3-none-any.whl |
C SDK | kxi-c-sdk 1.6.0 |
Java SDK | kxi-java-sdk 1.6.0 |
RT Bridge | rt-bridge 1.6.0 |
Infrastructure | kxi-terraform-1.6.0.tgz |
1.6.1
Release Date
2023-06-22
Improvements
- In the kdb Insights Database, a new routing parameter called scope has been introduced for routing query requests based on assembly. This parameter allows the user to specifically target individual assemblies for their query.
Artifacts
type | location |
---|---|
Enterprise | insights-1.6.1.tgz |
Operator | kxi-operator-1.6.0.tgz |
CLI | kxicli-1.6.0-py3-none-any.whl |
C SDK | kxi-c-sdk 1.6.0 |
Java SDK | kxi-java-sdk 1.6.0 |
RT Bridge | rt-bridge 1.6.0 |
Infrastructure | kxi-terraform-1.6.0.tgz |
1.6.0
Release Date
2023-06-20
Improvements
-
Support for air-gapped environments has been improved and both
kxi install run
andkxi install upgrade
support installing packaged charts (chart tgz’s) by passing the packaged chart name as an argument. If you use the default value ofkx-insights
for the chart repo, you can install it as follows:kxi install run --version 1.6.0 --filepath values.yaml insights-1.6.0.tgz
This argument also supports chart references, for example:
kxi install run --version 1.6.0 --filepath values.yaml <CHART_REPO>/insights
-
The kdb Insights Database can now work with or without an inventory file provided for an object storage tier. Using an inventory file for your object storage tier is still the recommended configuration, as this leverages caching and greatly improves performance when querying object storage data.
-
The kdb Insights Enterprise UI has an improved left hand menu. It can be minimized, and it is displayed alongside the Pipeline Operators list when the pipeline editor is active.
-
The kdb Insights Database now validates that primary keys for a partitioned table include the partition column. Previously, setting a primary key on a partitioned table that was not the partition column would error when queried.
-
Query timeout messages now indicate a reason for the timeout. Previously, query timeouts gave no reason, or said “timed out for unspecified reasons”. See queueing timeouts for more details.
-
Greater stability through the use of Kubernetes discovery between the Service Gateway and database processes.
-
Client-side pivot queries are now available in Views, with optional server-side computation for kdb+/q.
-
When calling
kxi install delete
the kdb Insights CLI now backs up assemblies before tearing them down.
Fixes
Expand to see the full detail of resolved issues here
Security
- Various CVEs have been remediated as part of this release.
Database
-
Resiliency improvements have been made to ensure the kdb Insights Database can recover from multiple days without end-of-day processing having completed. Previously, if more than 2 EODs were missed, it was possible to get a
misaligned
error on recovery. -
Validation has been added for assemblies to ensure that tiers' scheduled alignment time (snap value) are provided as the kdb+
time
type (19h
). Previously users could enter a timespan which would silently be ignored. Entering a timespan will now error.
Reliable Transport
- Kubernetes service account tokens are now automatically refreshed from disk. In previous versions these were not refreshed, causing failing requests to the Kubernetes control plane and pods had to be restarted.
Deprecation
CLI Install Commands
The following three arguments have been deprecated from kxi install setup
: chart-repo-name
, chart-repo-username
and chart-repo-url
.
To prevent breaking changes the arguments will still work, but they may be removed in a future version. Adding a chart repository is now decoupled from kxi
commands. The suggested approach when calling kxi install setup
is to add the chart repository directly using helm repo add
and to reference that repository by passing the chart reference as an argument.
If you use the default value of kx-insights
for the chart repo, you can install it as follows:
kxi install run --version 1.6.0 --filepath values.yaml
If you have used a non-default chart repo value then you can install with:
kxi install run --version 1.6.0 --filepath values <CHART_REPO>/insights
CLI Lifecycle Commands
The following commands have been removed. Please use the equivalents below to manage assemblies.
removed | supported command |
---|---|
kxi azure assembly backup | kxi assembly backup |
kxi azure assembly delete | kxi assembly teardown |
kxi azure assembly restore | kxi assembly deploy |
Third-party dependencies
kdb Insights Enterprise 1.6.0 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.6.0 release of the standalone infrastructure installation scripts.
Artifacts
type | location |
---|---|
Enterprise | insights-1.6.0.tgz |
Operator | kxi-operator-1.6.0.tgz |
CLI | kxicli-1.6.0-py3-none-any.whl |
C SDK | kxi-c-sdk 1.6.0 |
Java SDK | kxi-java-sdk 1.6.0 |
RT Bridge | rt-bridge 1.6.0 |
Infrastructure | kxi-terraform-1.6.0.tgz |
Known Issues
-
In 1.6.0 only, any code evaluated in the Python editor will throw a type error. To fix the Python editor for this issue you can run the following in a q editor:
\d .com_kx_edi python.run: { z: python.i.removeExtraIndents z; a: python.i.run[z; y]; $[x;$[y;python.i.truncate (),a;a];$[y;"";::]] }
-
When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
-
When writing Python code in the scratchpad, if you type
from kxi import sp
, it will automatically auto-correct tofrom kxi import super
. The workaround for this issue is as follows:- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit
super
back tosp
- Press the down arrow to go to the next line
-
On startup of pods, the following error might be observed after approximately three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault. -
A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.
To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using
last " " vs .z.l 7
. Save this to a variable$LICENSE_UUID
and run the commands belowklic license renew $LICENSE_UUID --version=latest env QLIC=$(pwd) klic license fetch $LICENSE_UUID
Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to
kxi-license
, set the$SECRET
variable if this isn’t correct.printf "data:\n license: %s\n" $(base64 -w0 kx.lic) > patch.yaml kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml
- When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. Therefore, there is a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.
- If a reader node is added to a pipeline that invokes
.qsp.read.fromFile
using chunking, then compilation fails.
- A known issue exists within the
kxi install rollback
command when rolling back to prior versions. When the target version was installed withkeycloak.initUser
enabled. It is expected thekeycloak-config-cli
job will fail. This is due to the rollback trying to revert that user's password and violating the password history policy.
The rollback failure will manifest as the failure of the keycloak-config-cli
job. This will create job pods as it fails and retries.
In order to proceed with the rollback it's recommended to delete kubectl delete job ${RELEASE_NAME}-keycloak-config-cli
.
A known issue exists during a reinstall using the CLI where the keycloak.initUser
is enabled. If the user's password has changed,
the reinstall will try to revert it back and be rejected by the password history policy.
This can be avoided by disabling the user import on reinstall using --import-users False
as below:
kxi install run --import-users False ..
Backward Compatibility
If performing a rollback from 1.6.0 to 1.5.x, a rollback chart is required. See Upgrading for more details.
1.5.3
Release Date
2023-06-19
Improvements
-
Resiliency improvements have been made to ensure the kdb Insights Database can recover from multiple days without end-of-day processing having completed. Previously, if more than 2 EODs were missed, it was possible to get a
misaligned
error on recovery. -
The kdb Insights Database can now work with or without an inventory file provided for an object storage tier. Using an inventory file for your object storage tier is still the recommended configuration, as this leverages caching and greatly improves performance when querying object storage data.
-
In cases where no data transformation is required, the kdb Insights Enterprise UI now allows Streams to be configured to write directly to the database. This reduces the amount of resources required for data ingestion and lowers operating costs.
Fixes
Expand to see the full detail of resolved issues here
Security
- Various CVEs have been remediated as part of this release.
User Interface
-
When creating a new database or renaming an existing database, you are now prevented from entering a database name longer than 24 characters. Previously, there was no restriction which could lead to name truncation.
-
The Database Schema Code View page now accepts all schema configuration options. Previously, it would not accept
blockSize
orisSharded
.
Reliable Transport
- Kubernetes service account tokens are now automatically refreshed from disk. In previous versions these were not refreshed, causing failing requests to the Kubernetes control plane and a need to restart your pods to recover.
Artifacts
type | location |
---|---|
Enterprise | insights-1.5.3.tgz |
Operator | kxi-operator-1.5.3.tgz |
CLI | kxicli-1.5.0-py3-none-any.whl |
C SDK | kxi-c-sdk 1.5.0 |
Java SDK | kxi-java-sdk 1.5.0 |
RT Bridge | rt-bridge 1.5.0 |
Infrastructure | kxi-terraform-1.5.1.tgz |
1.5.2
Release Date
2023-05-31
Improvements
-
In the kdb Insights Enterprise UI, when filtering queries in the Query page, distinct is now a supported aggregation type.
-
The
getData
API can now perform aggregations on foreign key columns.
Fixes
Expand to see the full detail of resolved issues here
- Kubernetes service account tokens are now automatically refreshed from disk. In previous versions these were not refreshed, causing failing requests to the Kubernetes control plane and pods had to be restarted.
Database
-
When deploying an assembly via the kdb Insights CLI, if ‘size’ is not specified for DAP instances, a default of 3 replicas will once again be applied. In 1.5.1, this default was increased to 4 to provide increased availability, but 3 has been restored as the standard to control costs. Users can configure a greater number of replicas if desired.
-
In the kdb Insights Enterprise UI, pipelines associated with a database now show in the status table. Previously, only the first pipeline would be shown.
-
In the kdb Insights Enterprise UI, the database schema overview now updates when changes are made to the database’s schema. Previously, this would only show the schema state when first viewed.
-
In the kdb Insights Enterprise UI, database documents no longer get marked as modified after a teardown or cleanup.
-
The database configuration UI now validates that the minimum rollover time is 1 minute. Previously, this could be set to 0.
Stream Processor
- Pipelines can be now be safely torn down without checkpoints having been cleared. Previously, if a teardown was followed by a restart, it is possible the pipeline would never run.
Artifacts
type | location |
---|---|
Enterprise | insights-1.5.2.tgz |
Operator | kxi-operator-1.5.2.tgz |
CLI | kxicli-1.5.0-py3-none-any.whl |
C SDK | kxi-c-sdk 1.5.0 |
Java SDK | kxi-java-sdk 1.5.0 |
RT Bridge | rt-bridge 1.5.0 |
Infrastructure | kxi-terraform-1.5.1.tgz |
1.5.0
Release Date
2023-05-09
New Features
UI
- The kdb Insights Enterprise UI has a new and improved look and user experience for viewing the details of your database(s). All relevant details about a database are now grouped together in a single summary page, giving users an intuitive view into the details and health of its components. Users can quickly see if any errors or warnings are present, and drill down into the details to investigate directly from the page. The details of your schema are conveniently presented for reference as well. To accompany this new interface, the term Assemblies has been simplified to Database in the UI. Assemblies remain the underlying concept tying components together, and are still referenced in documentation. Manual upgrade of old streams and schemas may be required. See upgrading entities in the UI for more details.
Queries
-
To protect against reserved word and column name conflicts, the way labels are passed to a query has changed. Labels should now be specified either as a dictionary, or with the prefix
label_
. The old label style is still supported in 1.5, but is deprecated and will be disabled by default in 1.6. See upgrade considerations on labels for more details. -
Distinct has been added to the available aggregations in
getData
for database queries.
Pipelines
-
Users can now use Python code in all areas of the pipeline UI, allowing for further flexibility in Python user experience with kdb Insights Enterprise. Previously, the addition of Python code to sections of pipeline UI were limited in scope to the function cells of the pipeline configuration forms.
-
The Stream Processor Python API can now leverage automatic schema conversion using a new function
kxi.sp.transform.schema
. -
Users can now use the Kafka Schema Registry in Python and the UI. See
kxi.sp.read.from_kafka
for Python, and the Kafka reader. -
String manipulation utilities have been added to the UI and the Stream Processor Python and q APIs, allowing users to selectively convert the string data to upper or lowercase representations for the purpose of data cleansing.
-
Non-UI deployments of Stream Processors now support pipeline replicas, providing enhanced resilience to deployed pipelines. Users can now choose to run their pipeline with replicas for high availability in the Stream Processor. This replaces the previous use of pipeline groups.
Reliable Transport
-
Reliable Transport now offers subscribers the ability to use dynamic client-side filtering, enabling users to control what parts of the data feed their subscribers need to receive. This can greatly improve ingestion performance by eliminating the need for subscribers to handle discarding of unnecessary data. See
Filtering
for more information. -
RT has introduced
logHistory
, a new API to assist in analysis and debugging. The API returns metadata about the RT log file history for logs that are no longer available. This includes:- The log file name.
- The time of the first message added to that log file.
- The position of the first message added to that log file.
See
Log File Meta Data
for more information.
Command Line Interface
-
A rollback command has been added to the CLI to simplify rolling back your installed instance of kdb Insights Enterprise after upgrading.
-
A
kxi user
subcommand has been added to the CLI to support managing users programmatically. This provides greater flexibility than using the keycloakinitUser
values at deploy time.
Standalone Installation
- Ability to control which components should be enabled (cert-manager,ingress-nginx, sharedfiles storage class). The available components are defined on the configuration file (
kxi-terraform.env
).
Improvements
-
Batch ingest now supports ingesting tables that have no symbols. Additionally, extra validation has been added to verify symbol integrity.
-
Batch ingest now supports sorting of columns.
-
The kdb Insights Database gateway now supports returning responses that are greater than 2GB for IPC requests. See routing configuration for more details.
-
Database schemas in the UI can now be modified for assemblies that have been stopped. See schema modification.
-
Requests that supply a log correlator (logCorr) will now see that correlator in the Service Gateway logs. Previously a random correlator was used in the Service Gateway logs.
-
Users can now utilize backtick syntax when defining aggregations in Views, e.g.
`c1`avg`price
. -
Pipeline editors can now be resized vertically.
-
Labels for improved accessibility have been added to the Pipelines screen.
-
Added significant improvements for accessibility using keyboard navigation and/or screen readers in the UI.
-
Users can exit a text area in a code block on the Query page by first pressing "CTRL+M" before pressing the "Tab" key.
-
The entity tree menu button can now be opened when hovering your mouse over the entity tree. Arrows can then be used to navigate.
-
"Code View" on the Schemas page can now be closed with the "Esc" key.
-
The Views and Pipeline pages now allow draggable elements to be inserted/dragged onto the page with the use of your keyboard.
-
- Kubernetes 1.25 is now supported and the default version for standalone installations.
- Ensure the latest AWS CLI is installed on the docker image used for standalone installations
Fixes
Expand to see the full detail of resolved issues here
Database
-
Additional historical tiers that used object storage in the UI now allow dashes and spaces in file names. Previously they would error for dashes and spaces.
-
Migrating tiers to object storage could fail with the eventual error
SMDBM Migration discontinued due to failure
if multiple threads were used. -
New validation has been added to prevent accidentally adding a object storage tier to a database that does not already have one. This would previously error at EOD with a file not found error.
-
Schema conversions are now more robust on network based file systems (ex. Rook Ceph). Previously, this could result in a
EXDEV. OS Reports: Invalid cross-device link error
in the Storage Manager. -
Changes to disk sort columns in a schema that would be incompatible with the current disk sort will now error with a clear message at deploy time. Previously, this would error with a
u-fail
error either at an EOI or EOD. -
Batch ingest can now gracefully handle scenarios where there is no data provided in the session. Previously, this would result in the following error:
Error reporting to licensing server: Value must be a long
, leaving the ingest pipeline stuck in aRUNNING
state. -
When removing a symbol column from a table, the SM should be able to properly handle its removal as well as the sym file.
-
If an in-memory sort is applied to a column in a schema, it is now sorted on arrival in the RDB. Previously, if data arrived out of order with this set, an
s-fail
error would be thrown. -
In the Storage Manager metrics, the
kxi_sm_hdb_date_records
metric no longer reports an empty value at startup, which resulted in false alerts about the HDB being offline. -
The Storage Manager now double buffers its state table in order to protect against corruption.
-
Rapid successive queries using
qsql
, such as those from Views in the kdb Insights Enterprise UI, no longer have the potential to return query results out of order. -
The
kxi_sg_pending
metric would previously be decremented twice when a connection closed, resulting in negative pending values. -
The Service Gateways were creating duplicate connections to Resource Coordinators while they were waiting to come online. This was resulting in a large number of redundant connections being opened between the services. In large deployments, this could result in
conn
errors in the Resource Coordinator. -
RDBs and HDBs that require a dependency on the IDB will now error at startup if one is not specified as part of the mount configuration.
User Interface
-
Previously, a database would show a loading spinner if one of its pipelines had reached a ‘finished’ state while its other components were in a ‘ready' state. This has now been rectified and the database will display a ‘ready’ state in this case.
-
An issue with pasting into text fields in a pipeline node’s configuration has been fixed. Previously, the node may be been duplicated.
Stream Processor
-
Pipeline names are now properly displayed in the Running Pipelines section of the Overview page. Previously, these were displaying the pipeline ID and were not clickable for viewing further information.
-
Pipelines deployed via as part of a Database now properly set Kubernetes secrets. Previously, secrets were only being set properly for standalone pipelines.
-
The presence of a UTF-8 byte-order mark at the beginning of CSV files will now be automatically removed from data when acted on by the Stream Processor CSV decoder.
-
Expression readers now return the result of Python lambdas.
-
Window nodes in the Stream Processor can now accept dictionaries as input batches. Previously, incoming dictionaries would be merged with input batch in the buffer, instead of appending to a buffered list of dictionaries.
-
The python
write.to_database
node previously converted assembly names to booleans. This is now resolved, and also fully supports the direct write functionality. -
The Stream Processor's Protobuf decoder no longer ignores the value of
As List
when a pipeline is deployed from the UI. Previously, this value was being overwritten. -
The Stream Processors Protocol Buffer Encoder node now properly saves the selected Payload Type.
-
The Stream Processor Database Reader now supports SQL statement queries. Previously, this node had erroneously suggested the use of qsql statements for querying which was unsupported.
-
The default
onFinish
functions for the apply node have been updated to have the correct parameters and examples of flushing a buffer. -
When using directWrite functionality in
.qsp.write.toDatabase
, table names as strings are now supported. Previously, this had to be a symbol. -
Copy/pasting nodes within a pipeline document while other pipeline documents are open in the background no longer produces duplicates. Previously, the copy/paste operation was being applied to both the intended pipeline as well as the unintended pipelines, resulting in nodes disappearing due to shared names/ids.
-
Security vulnerabilities patched in the Stream Processor for the following CVEs: CVE-2022-41910, CVE-2022-41902, CVE-2022-41900, CVE-2022-41880.
-
Query sessions had previously set error trapping to
1
within the q session, resulting in users being unable to interact with their query session. This behaviour manifested in users being unable to interact with their query sessions until they were manually restarted.
Command Line Interface
- With previous versions of the kdb Insights CLI, the kxi install list-versions command would return an outdated list of versions. This has now been resolved and the list will be updated on each invocation.
Standalone Installation
- Fix Rook-Ceph OSD not getting assigned if underlying node changes
Third-party dependencies
kdb Insights Enterprise 1.5.0 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.5.0 release of the standalone infrastructure installation scripts.
Artifacts
type | location |
---|---|
Enterprise | insights-1.5.0.tgz |
Operator | kxi-operator-1.5.0.tgz |
CLI | kxicli-1.5.0-py3-none-any.whl |
C SDK | kxi-c-sdk 1.5.0 |
Java SDK | kxi-java-sdk 1.5.0 |
RT Bridge | rt-bridge 1.5.0 |
Infrastructure | kxi-terraform-1.5.0.tgz |
Upgrade notes
Upgrading existing assemblies
With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly from 1.3.x to any newer release, you must delete the RT PVCs.
Causes duplicate data
Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.
- Stop all external publishers to kdb Insights Enterprise
- Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data
-
Backup any assemblies that may be running
$ kxi assembly backup --filepath assembly-state-file.yaml Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml
-
Teardown assemblies named in the output of the backup command
$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig Tearing down assembly dfx-assembly Are you sure you want to teardown dfx-assembly [y/N]: y Waiting for assembly to be torn down [------------------------------------] 0% $ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig Tearing down assembly iot-assembly Are you sure you want to teardown iot-assembly [y/N]: y Waiting for assembly to be torn down [------------------------------------] 0%
-
Delete the RT PVCs
kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>
-
Carry out the upgrade
-
Restore assemblies
kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig
Restriction on upgrading to older versions
The CLI will not allow you to "upgrade" to an older version of kdb Insights Enterprise. Please use the rollback command to revert to a previous version.
Known Issues
-
On startup of pods, the following error might be observed after approximately three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault. -
A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.
To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using
last " " vs .z.l 7
. Save this to a variable$LICENSE_UUID
and run the commands belowklic license renew $LICENSE_UUID --version=latest env QLIC=$(pwd) klic license fetch $LICENSE_UUID
Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to
kxi-license
, set the$SECRET
variable if this isn’t correct.printf "data:\n license: %s\n" $(base64 -w0 kx.lic) > patch.yaml kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml
- Publishing of a single message larger than 1GB is not supported.
- Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
- When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. Therefore, there is a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.
-
If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
-
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.
-
When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
-
When writing Python code in the scratchpad, if you type
from kxi import sp
, it will automatically auto-correct tofrom kxi import super
. The workaround for this issue is as follows:- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit
super
back tosp
- Press the down arrow to go to the next line
-
Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
-
If a reader node is added to a pipeline that invokes
.qsp.read.fromFile
using chunking, then compilation fails.
- A known issue exists within the
kxi install rollback
command when rolling back to prior versions. When the target version was installed withkeycloak.initUser
enabled. It is expected thekeycloak-config-cli
job will fail. This is due to the rollback trying to revert that user's password and violating the password history policy.
The rollback failure will manifest as the failure of the keycloak-config-cli
job. This will create job pods as it fails and retries.
In order to proceed with the rollback it's recommended to delete kubectl delete job ${RELEASE_NAME}-keycloak-config-cli
.
A known issue exists during a reinstall using the CLI where the keycloak.initUser
is enabled. If the user's password has changed,
the reinstall will try to revert it back and be rejected by the password history policy.
This can be avoided by disabling the user import on reinstall using --import-users False
as below:
kxi install run --import-users False ..
Backward Compatibility
If you are upgrading from a version earlier than 1.5.0 you must also read each of the relevant release notes below to ensure you know all the changes that will affect the upgrade.
1.4.5
Release for kdb Insights Enterprise.
Release Date
2023-04-26
Improvements
[IMPROVEMENT]: If publishing a message larger than 1GB, the publisher will now receive a messsage too large
error. Users can catch the error (need more clear detail) to avoid getting an error at runtime. This would previously cause downstream subscriber processes to run out of memory during processing.
[IMPROVEMENT]: SQL queries that filter by time but do not explicitly select the time column are now supported. This would previously error.
Fixes
[FIX]: Fixed an issue where rapid successive queries using qsql
such as those from Views in the kdb Insights UI could return query results out of order.
[FIX]: Schema conversions are now more robust on network based file systems (ex. Rook Ceph). Previously, this could result in a EXDEV. OS Reports: Invalid cross-device link error
in the Storage Manager.
[FIX]: Window nodes in the Stream Processor can now accept dictionaries as input batches. Previously, incoming dictionaries would be merged with input batch in the buffer, instead of appending to a buffered list of dictionaries.
Third-party dependencies
kdb Insights Enterprise 1.4 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.4 release of the standalone infrastructure installation scripts.
Artifacts
type | location |
---|---|
Enterprise | insights-1.4.5.tgz |
Operator | kxi-operator-1.4.2.tgz |
CLI | kxicli-1.4.2-py3-none-any.whl |
C SDK | kxi-c-sdk 1.4.0 |
Java SDK | kxi-java-sdk 1.4.0 |
RT Bridge | rt-bridge 1.4.0 |
Infrastructure | kxi-terraform-1.4.1.tgz |
Upgrade notes
Upgrading existing assemblies
With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly created using a version prior to 1.4.0, you must delete the RT PVCs.
Causes duplicate data
Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.
- Stop all external publishers to kdb Insights Enterprise
- Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data
-
Backup any assemblies that may be running
$ kxi assembly backup --filepath assembly-state-file.yaml Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml
-
Teardown assemblies named in the output of the backup command
$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig Tearing down assembly dfx-assembly Are you sure you want to teardown dfx-assembly [y/N]: y Waiting for assembly to be torn down [------------------------------------] 0% $ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig Tearing down assembly iot-assembly Are you sure you want to teardown iot-assembly [y/N]: y Waiting for assembly to be torn down [------------------------------------] 0%
-
Delete the RT PVCs
kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>
-
Carry out the upgrade
-
Restore assemblies
kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig
Licensing
A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.
To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7
. Save this to a variable $LICENSE_UUID
and run the commands below.
klic license renew $LICENSE_UUID --version=latest
env QLIC=$(pwd) klic license fetch $LICENSE_UUID
Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license
, set the $SECRET
variable if this isn’t correct.
printf "data:\n license: %s\n" $(base64 -w0 kx.lic) > patch.yaml
kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml
User import
The initUser
will be automatically imported on initial install but not on upgrade. This prevents the initUser
password from being reset on upgrade.
A new value has been added to allow you to change this behavior from the default.
keycloak:
importUsers: false
Monitoring
We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes.
If you wish to continue with the higher frequency you can take one of the following steps:
- Update the interval field within their charts.
```yaml
metrics:
serviceMonitor:
interval: "30s"
```
- Override the chart values using the global value within insights/values.yaml.
```yaml
global:
metrics:
serviceMonitor:
interval: "30s"
```
- Update your assembly yaml, to override the default.
```yaml
spec:
metrics:
enabled: true
serviceMonitor:
enabled: true
interval: 10s
```
- Update your default within the Assembly Resource - This is generated using the insights/values.yaml.
```yaml
kxi-operator:
config:
sidecar:
metrics:
serviceMonitor:
interval: 30s
```
Default password policy
The default password policy has been updated to include the following rule:
- Not one of the previous 24 passwords
If you are upgrading from an earlier version and set importUsers: true
, the password for initUser
must be updated to a new, valid password in the values file before upgrade.
Default Keycloak Credentials
As of 1.3.0 the Keycloak initUser
and initClient
credentials are no longer defaulted within the kdb Insights values.yaml
. See here for more details.
Downgrading to earlier versions
Downgrading to earlier versions was previously supported by executing kxi install upgrade
. This is no longer supported. To install an earlier version, execute the rollback steps.
Schema Configuration Changes
The field updTsCol
is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.
Known Issues
- Publishing of a single message larger than 1GB is not supported.
- On startup of pods, the following error might be observed after approximately three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.
-
Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
-
Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.
When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.
-
If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
-
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.
-
When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
-
When writing Python code in the scratchpad, if you type
from kxi import sp
, it will automatically auto-correct tofrom kxi import super
. The workaround for this issue is as follows:- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit
super
back tosp
- Press the down arrow to go to the next line
- If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.
The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.
"[] Failed to compare saved assembly to deployed version, error: type"
-
Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
-
If a reader node is added to a pipeline that invokes
.qsp.read.fromFile
using chunking, then compilation fails.
- When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of
colAlias;aggFn;col
. Symbols lists are currently not supported.
-
Screen readers will not read out node information represented in a pop-up when hovered over, and may freeze when attempting to read a large amount of content.
-
Unable to navigate up and down cells and read all table content when navigating using only a keyboard.
- The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration
values.yaml
.
keycloak:
postgresql:
primary:
resources:
limits:
cpu: 2000m
requests:
cpu: 50m
1.4.4
Release for kdb Insights Enterprise.
Release Date
2023-04-20
Improvements
[IMPROVEMENT]: The kdb Insights Database now supports a REST interface for loading historical datasets into an existing database. You can now perform a historical batch ingest using a simple REST API. For more details, see batch ingestion.
[IMPROVEMENT]: Query requests that supply a log correlator (logCorr
) will now see that correlator in the Service Gateway logs, allowing for easier traceability in query execution and an improved debugging experience. Previously a random correlator was used in the Service Gateway logs.
[IMPROVEMENT]: Added significant improvements for accessibility using keyboard navigation and/or screen readers in the UI.
-
Users can exit a text area in a code block on the Query page by first pressing "CTRL+M" before pressing the "Tab" key.
-
The entity tree menu button can now be opened when hovering your mouse over the entity tree. Arrows can then be used to navigate.
-
"Code View" on the Schemas page can now be closed with the "Esc" key.
-
The Views and Pipeline pages now allow draggable elements to be inserted/dragged onto the page with the use of your keyboard.
Fixes
[FIX]: Security vulnerabilities patched in the Stream Processor for the following CVEs: CVE-2022-41910, CVE-2022-41902, CVE-2022-41900, CVE-2022-41880.
[FIX]: Additional historical tiers that use object storage in the UI now allow dashes and spaces in file names. Previously they would error for dashes and spaces.
[FIX]: Migrating tiers to object storage could fail with the eventual error SMDBM Migration discontinued due to failure
if multiple threads were used.
[FIX]: If an SM client cannot connect to SM after a number of retries, it will now restart the pod to try to automatically recover. This resolves connection issues in DAPs when there are network connectivity issues.
[FIX]: Reading of files from Azure Storage larger than ~1MB now successfully writes to a kdb Insights database. Previously this was erroring due to an issue in the chunking of larger files.
[FIX]: The Service Gateway was creating duplicate connections to Resource Coordinators while they were waiting to come online. This was resulting in a large number of redundant connections being opened between the services. In large deployments, this could result in connection errors in the Resource Coordinator.
[FIX]: Expression readers now return the result of Python lambdas.
[FIX]: After upgrading kdb Insights Enterprise from 1.3.x to 1.4.x, clicking a node in a previously-saved pipeline in the UI no longer causes the function definition to flicker and disappear.
[FIX]: Fixed issue with 1.4.0 kxi-terraform scripts, where Terragrunt config was incompatible and was causing deployment failures on AWS.
Third-party dependencies
kdb Insights Enterprise 1.4 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.4 release of the standalone infrastructure installation scripts:target="_blank"}.
Artifacts
type | location |
---|---|
Enterprise | insights-1.4.4.tgz |
Operator | kxi-operator-1.4.2.tgz |
CLI | kxicli-1.4.2-py3-none-any.whl |
C SDK | kxi-c-sdk 1.4.0 |
Java SDK | kxi-java-sdk 1.4.0 |
RT Bridge | rt-bridge 1.4.0 |
Infrastructure | kxi-terraform-1.4.1.tgz |
Upgrade notes
Upgrading existing assemblies
With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly created using a version prior to 1.4.0, you must delete the RT PVCs.
Causes duplicate data
Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.
- Stop all external publishers to kdb Insights Enterprise
- Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data
-
Backup any assemblies that may be running
$ kxi assembly backup --filepath assembly-state-file.yaml Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml
-
Teardown assemblies named in the output of the backup command
$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig Tearing down assembly dfx-assembly Are you sure you want to teardown dfx-assembly [y/N]: y Waiting for assembly to be torn down [------------------------------------] 0% $ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig Tearing down assembly iot-assembly Are you sure you want to teardown iot-assembly [y/N]: y Waiting for assembly to be torn down [------------------------------------] 0%
-
Delete the RT PVCs
kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>
-
Carry out the upgrade
-
Restore assemblies
kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig
Licensing
A number of stability issues were identified around license checking. These could cause adverse performance in some circumstances so it is recommended to upgrade your license to prevent these issues.
To do this, grab your license UUID from the System Information dialogue in the kdb Insights Enterprise UI or from a running process using last " " vs .z.l 7
. Save this to a variable $LICENSE_UUID
and run the commands below.
klic license renew $LICENSE_UUID --version=latest
env QLIC=$(pwd) klic license fetch $LICENSE_UUID
Once the license is fetched you can update your license secret using the command below. Below commands default the license secret name to kxi-license
, set the $SECRET
variable if this isn’t correct.
printf "data:\n license: %s\n" $(base64 -w0 kx.lic) > patch.yaml
kubectl patch secret ${SECRET:-kxi-license} --patch-file=patch.yaml
User import
The initUser
will be automatically imported on initial install but not on upgrade. This prevents the initUser
password from being reset on upgrade.
A new value has been added to allow you to change this behavior from the default.
keycloak:
importUsers: false
Monitoring
We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes.
If you wish to continue with the higher frequency you can take one of the following steps:
- Update the interval field within their charts.
```yaml
metrics:
serviceMonitor:
interval: "30s"
```
- Override the chart values using the global value within insights/values.yaml.
```yaml
global:
metrics:
serviceMonitor:
interval: "30s"
```
- Update your assembly yaml, to override the default.
```yaml
spec:
metrics:
enabled: true
serviceMonitor:
enabled: true
interval: 10s
```
- Update your default within the Assembly Resource - This is generated using the insights/values.yaml.
```yaml
kxi-operator:
config:
sidecar:
metrics:
serviceMonitor:
interval: 30s
```
Default password policy
The default password policy has been updated to include the following rule:
- Not one of the previous 24 passwords
If you are upgrading from an earlier version and set importUsers: true
, the password for initUser
must be updated to a new, valid password in the values file before upgrade.
Default Keycloak Credentials
As of 1.3.0 the Keycloak initUser
and initClient
credentials are no longer defaulted within the kdb Insights values.yaml
. See here for more details.
Downgrading to earlier versions
Downgrading to earlier versions was previously supported by executing kxi install upgrade
. This is no longer supported. To install an earlier version, execute the rollback steps.
Schema Configuration Changes
The field updTsCol
is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.
Known Issues
- On startup of pods, the following error might be observed after approximately three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault.
-
Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
-
Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.
- When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.
-
If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
-
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.
-
When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
-
When writing python code in the scratchpad, if you type
from kxi import sp
, it will automatically auto-correct tofrom kxi import super
. The workaround for this issue is as follows:- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit
super
back tosp
- Press the down arrow to go to the next line
- If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.
- The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.
"[] Failed to compare saved assembly to deployed version, error: type"
-
Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
-
If a reader node is added to a pipeline that invokes
.qsp.read.fromFile
using chunking, then compilation fails.
- When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of
colAlias;aggFn;col
. Symbols lists are currently not supported.
-
Screen readers will not read out node information represented in a pop-up when hovered over, and may freeze when attempting to read a large amount of content.
-
Unable to navigate up and down cells and read all table content when navigating using only a keyboard.
- The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration
values.yaml
.
keycloak:
postgresql:
primary:
resources:
limits:
cpu: 2000m
requests:
cpu: 50m
1.4.3
Release Date
2023-03-31
Improvements
[IMPROVEMENT] We have reduced the frequency that Prometheus will scrape the metrics from a sidecar. You will be affected if you are refreshing/polling monitoring dashboards at a rate below 2 minutes.
If you wish to continue with the higher frequency you can take one of the following steps:
- Update the interval field within their charts.
```yaml
metrics:
serviceMonitor:
interval: "30s"
Assemblies
```
-
Update your assembly yaml, to override the default.
```yaml spec: metrics: enabled: true serviceMonitor: enabled: true interval: 10s ```
-
Update your default within the Assembly Resource - This is generated using the insights/values.yaml.
```yaml kxi-operator: config: sidecar: metrics: serviceMonitor: interval: 30s ```
Fixes
[FIX]: Service Gateway: fixed remote code injection vulnerability in Snake YAML CVE-2022-1471
[FIX]: Queries to the kdb Insights Database now have a default timeout of 60s (previously 30s). This allows for longer running queries to be issued from query clients, such as the scratchpad.
[FIX]: Additional historical tiers in the UI now allow for a retention period to be set on the HDB tier. Previously this was erroring due to a validation issue.
[FIX]: In the UI, the real-time database rollover tier now correctly checks that rollover is less than the interval tier.
[FIX]: Requests to object storage (Azure Storage, AWS S3, Google Cloud Storage) will now backoff and retry if they get a throttle request from the vendor. Jitter will be introduced to the retry to ensure that replicated processes do not all retry at the same time.
[FIX]: Setting a column type to Any now correctly uses an anymap type. This was previously reset to an unselected value.
[FIX]: The REST Server's heartbeat logs are now set to TRACE to reduce log output (previously they were logged as INFO logs).
[FIX]: Pasting into a pipeline node’s configuration no longer duplicates the node.
[FIX]: Resolved issue where some pipelines with merge nodes would error with 'missing edge'.
[FIX]: Comments can now be added to the last line of the Scratchpad without causing it to error.
[FIX]: The Scratchpad now has more memory allocated for ad-hoc queries in the kdb Insights Enterprise query panel.
[FIX]: Stream Processor file reader nodes (Azure, AWS S3, File and Google storage) would only process the first file presented even when multiple files were present.
[FIX]: Object storage tiers can now be added to a database via the kdb Insights Enterprise UI.
Third-party dependencies
kdb Insights Enterprise 1.4.3 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.4.3 release of the standalone infrastructure installation scripts.
Artifacts
type | location |
---|---|
Upgrade notes
Upgrading existing assemblies
With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.
Causes duplicate data
Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.
- Stop all external publishers to kdb Insights Enterprise
- Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data
-
Backup any assemblies that may be running
$ kxi assembly backup --filepath assembly-state-file.yaml Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml
-
Teardown assemblies named in the output of the backup command
$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig Tearing down assembly dfx-assembly Are you sure you want to teardown dfx-assembly [y/N]: y Waiting for assembly to be torn down [------------------------------------] 0% $ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig Tearing down assembly iot-assembly Are you sure you want to teardown iot-assembly [y/N]: y Waiting for assembly to be torn down [------------------------------------] 0%
-
Delete the RT PVCs
kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>
-
Carry out the upgrade
-
Restore assemblies
kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig
User import
The initUser
will be automatically imported on initial install but not on upgrade. This prevents the initUser
password from being reset on upgrade.
A new value has been added to allow you to change this behavior from the default.
keycloak:
importUsers: false
Default password policy
The default password policy has been updated to include the following rule:
- Not one of the previous 24 passwords
If you are upgrading from an earlier version and set importUsers: true
, the password for initUser
must be updated to a new, valid password in the values file before upgrade.
Default Keycloak Credentials
As of 1.3.0 the Keycloak initUser
and initClient
credentials are no longer defaulted within the kdb Insights values.yaml
. See here for more details.
Downgrading to earlier versions
Downgrading to earlier versions was previously supported by executing kxi install upgrade
. This is no longer supported. To install an earlier version, execute the rollback steps.
Schema Configuration Changes
The field updTsCol
is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.
Known Issues
-
On startup of pods, the following error might be observed after approximately three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault. -
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes
unable to flush accounting logs
. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.
-
Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
-
Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.
-
Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.
When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.
-
If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
-
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.
-
When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
-
When writing python code in the scratchpad, if you type
from kxi import sp
, it will automatically auto-correct tofrom kxi import super
. The workaround for this issue is as follows:- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit
super
back tosp
- Press the down arrow to go to the next line
- If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.
The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.
"[] Failed to compare saved assembly to deployed version, error: type"
-
Please ensure all pipelines have different names. Two pipelines with the same name may not be deploy correctly.
-
Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
-
If a reader node is added to a pipeline that invokes
.qsp.read.fromFile
using chunking, then compilation fails.
When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col
. Symbols lists are currently not supported.
The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml
.
keycloak:
postgresql:
primary:
resources:
limits:
cpu: 2000m
requests:
cpu: 50m
1.4.2
Release for kdb Insights Enterprise.
Release Date
2023-03-17
Fixes
[FIX] Resource utilization in the UI Scratchpad has been greatly improved. This fix along with a larger default memory allocation provide greater Scratchpad process stability.
Third-party dependencies
kdb Insights Enterprise 1.4.2 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts.
Artifacts
type | location |
---|---|
Enterprise | insights-1.4.2.tgz |
Operator | kxi-operator-1.4.0.tgz |
CLI | kxicli-1.4.1-py3-none-any.whl |
C SDK | kxi-c-sdk 1.4.0 |
Java SDK | kxi-java-sdk 1.4.0 |
RT Bridge | rt-bridge 1.4.0 |
Infrastructure | kxi-terraform-1.4.0.tgz |
Upgrade notes
Upgrading existing assemblies
With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.
Causes duplicate data
Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.
- Stop all external publishers to kdb Insights Enterprise
- Wait for all published messages to be available for querying from the database; this will reduce (but not eliminate) the amount of duplicate data
-
Backup any assemblies that may be running
$ kxi assembly backup --filepath assembly-state-file.yaml Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml
-
Teardown assemblies named in the output of the backup command
$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig Tearing down assembly dfx-assembly Are you sure you want to teardown dfx-assembly [y/N]: y Waiting for assembly to be torn down [------------------------------------] 0% $ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig Tearing down assembly iot-assembly Are you sure you want to teardown iot-assembly [y/N]: y Waiting for assembly to be torn down [------------------------------------] 0%
-
Delete the RT PVCs
kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>
-
Carry out the upgrade
-
Restore assemblies
kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig
User import
The initUser
will be automatically imported on initial install but not on upgrade. This prevents the initUser
password from being reset on upgrade.
A new value has been added to allow you to change this behavior from the default.
keycloak:
importUsers: false
Default password policy
The default password policy has been updated to include the following rule:
- Not one of the previous 24 passwords
If you are upgrading from an earlier version and set importUsers: true
, the password for initUser
must be updated to a new, valid password in the values file before upgrade.
Default Keycloak Credentials
As of 1.3.0 the Keycloak initUser
and initClient
credentials are no longer defaulted within the kdb Insights values.yaml
. See here for more details.
Downgrading to earlier versions
Downgrading to earlier versions was previously supported by executing kxi install upgrade
. This is no longer supported. To install an earlier version, execute the rollback steps.
Schema Configuration Changes
The field updTsCol
is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.
Known Issues
-
On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault. -
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes
unable to flush accounting logs
. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.
-
Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
-
Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.
-
Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.
When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.
-
If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
-
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.
-
When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
-
When writing python code in the scratchpad, if you type
from kxi import sp
, it will automatically auto-correct tofrom kxi import super
. The workaround for this issue is as follows:- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit
super
back tosp
- Press the down arrow to go to the next line
- If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.
The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.
"[] Failed to compare saved assembly to deployed version, error: type"
-
When creating a pipeline the user interface does not stop you from creating it with the same name as an existing pipeline, however two pipelines with the same name may not be deploy correctly. Please ensure all pipelines have different names.
-
Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
-
If a reader node is added to a pipeline that invokes
.qsp.read.fromFile
using chunking, then compilation fails.
When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col
. Symbols lists are currently not supported.
The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml
.
keycloak:
postgresql:
primary:
resources:
limits:
cpu: 2000m
requests:
cpu: 50m
1.4.1
Release for kdb Insights Enterprise.
Release Date
2023-03-14
Fixes
[FIX] In 1.3.3 and 1.4.0, if the Storage Manager falls behind on EODs for more than a full day, IDB links could be written incorrectly when late data was present. If this issue is encountered, manual data changes are required to recover the database including backup of the stream logs from the date of the failure. Please contact KX Support for help with recovery.
This issue is characterized by a log statement from the Storage Manager as follows: "Unexpected condition: EOXa watermark date YYYY.MM.DD is less than HDB stream date YYYY.MM.DD, setting it to match HDB stream date". The IDB may also start reporting "No such file or directory" on ordinal partitions.
Third-party dependencies
kdb Insights Enterprise 1.4.1 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts.
Artifacts
type | location |
---|---|
Enterprise | insights-1.4.1.tgz |
Operator | kxi-operator-1.4.0.tgz |
CLI | kxicli-1.4.1-py3-none-any.whl |
C SDK | kxi-c-sdk 1.4.0 |
Java SDK | kxi-java-sdk 1.4.0 |
RT Bridge | rt-bridge 1.4.0 |
Infrastructure | kxi-terraform-1.4.0.tgz |
Upgrade notes
Upgrading existing assemblies
With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.
Causes duplicate data
Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.
- Stop all external publishers to kdb Insights Enterprise
- Wait for all published messages to be available for querying from the database - this will reduce (but not eliminate) the amount of duplicate data
-
Backup any assemblies that may be running
$ kxi assembly backup --filepath assembly-state-file.yaml Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml
-
Teardown assemblies named in the output of the backup command
$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig Tearing down assembly dfx-assembly Are you sure you want to teardown dfx-assembly [y/N]: y Waiting for assembly to be torn down [------------------------------------] 0% $ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig Tearing down assembly iot-assembly Are you sure you want to teardown iot-assembly [y/N]: y Waiting for assembly to be torn down [------------------------------------] 0%
-
Delete the RT PVCs
kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>
-
Carry out the upgrade
-
Restore assemblies
kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig
User import
The initUser
will be automatically imported on initial install but not on upgrade. This prevents the initUser
password from being reset on upgrade.
A new value has been added to allow you to change this behavior from the default.
keycloak:
importUsers: false
Default password policy
The default password policy has been updated to include the following rule:
- Not one of the previous 24 passwords
If you are upgrading from an earlier version and set importUsers: true
, the password for initUser
must be updated to a new, valid password in the values file before upgrade.
Default Keycloak Credentials
As of 1.3.0 the Keycloak initUser
and initClient
credentials are no longer defaulted within the kdb Insights values.yaml
. See here for more details.
Downgrading to earlier versions
Downgrading to earlier versions was previously supported by executing kxi install upgrade
. This is no longer supported. To install an earlier version, execute the rollback steps.
Schema Configuration Changes
The field updTsCol
is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.
Known Issues
-
On startup of pods, the following error might be observed after approximately three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault. -
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes
unable to flush accounting logs
. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.
-
If end-of-day processing is interrupted and remains incomplete until the next EOD period, this prevents further EODs from completing and requires manual recovery.
-
Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
-
Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.
-
Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.
When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.
-
If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
-
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.
-
When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
-
When writing Python code in the scratchpad, if you type
from kxi import sp
, it will automatically auto-correct tofrom kxi import super
. The workaround for this issue is as follows:- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit
super
back tosp
- Press the down arrow to go to the next line
- If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.
The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.
"[] Failed to compare saved assembly to deployed version, error: type"
-
Please ensure all pipelines have different names. Two pipelines with the same name may not be deploy correctly.
-
Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
-
If a reader node is added to a pipeline that invokes
.qsp.read.fromFile
using chunking, then compilation fails.
When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col
. Symbols lists are currently not supported.
The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml
.
keycloak:
postgresql:
primary:
resources:
limits:
cpu: 2000m
requests:
cpu: 50m
1.4.0
Release for kdb Insights Enterprise.
Release Date
2023-03-03
New Features
Introducing Pythonic pipeline development in the UI, dynamic logging notifications providing direct access to root cause of underlying issues, an enhanced user experience for database configuration and additional security enhancements.
[NEW] Pipelines
-
You can now use Python in the pipeline builder in the UI to perform aggregations and analytics.
-
Ingest large batches of data directly into the historical database has been moved out of beta.
- Use fewer resources. Memory and disk resources greatly reduced during ingest.
- Select the
Write Direct to HDB
field in the Database Writer in the UI, or use the q and Python APIs.
[NEW] User Interface
-
Improvements to the observability user experience with two new icons in top right hand-side of the UI banner:
-
A notification icon shows a count of new log messages. This ensures you are notified of any new error messages wherever you are in the UI. You can also click on the icon to show more details of the message, these are displayed in the right-hand third of the screen.
-
A settings icon enables you to update the application logging level for future log records written by the kdb Insights Enterprise components.
-
-
You can now share the URL of a View by selecting the view on the left-hand toolbar, right-clicking and choosing the new
Share
button. This URL just shares the content of the View and not the toolbars (e.g. it does not show the left-hand toolbar or the Design/Preview panel at the top).
[NEW] Databases
A new version of our Database Configuration page is now available, under a beta flag, that provides an enhanced user experience. It can be toggled on and off from the Settings option in the main toolbar. We recommend that you use the existing Database Configuration age for all running databases, but please try the new user interface out and give us your feedback.
[NEW] Packages
The user interface now allows you to configure which Packages and their associated Custom APIs are used within individual databases, this had previously only been possible through kdb CLI deployments.
[NEW] Security Improvements
- Keycloak has been updated to version 19. This impacts the following:
- The Keycloak Admin console has been changed
- Logout screen provides a redirection to the login page
Improvements
Keycloak realm
In this release, the handling of the Keycloak realm has been improved. This realm contains the role definitions, users and clients. Previously the realm was not being imported by default on install. The role definitions will always be imported as part of an install or upgrade now. The users and clients will only be imported by default on install (kxi install
), and not on upgrade.
To make changes to the initClient
and initUser
on upgrade, you will need to tell the application to import them using the setting below.
keycloak:
importUsers: true
Reliable Transport
The kdb Insights Enterprise transport layer has adopted a version of Raft called NuRaft. This improves the fault tolerance and performance of kdb Insights Enterprise. NuRaft is an open source version of Raft. Migrating to NuRaft also provides kdb Insights Enterprise with additional capabilities for future enhancements to Reliable Transport.
Third-party dependencies
kdb Insights Enterprise 1.4.0 standalone install supports the following versions of third-party dependencies:
These versions are used in the 1.4.0 release of the standalone infrastructure installation scripts .
Artifacts
type | location |
---|---|
Enterprise | insights-1.4.0.tgz |
Operator | kxi-operator-1.4.0.tgz |
CLI | kxicli-1.4.0-py3-none-any.whl |
C SDK | kxi-c-sdk 1.4.0 |
Java SDK | kxi-java-sdk 1.4.0 |
RT Bridge | rt-bridge 1.4.0 |
Infrastructure | kxi-terraform-1.4.0.tgz |
Upgrade notes
Upgrading existing assemblies
With the adoption of NuRaft by the Reliable Transport (RT) service, if you are upgrading an existing assembly to version 1.4.0, you must delete the RT PVCs.
Causes duplicate data
Due to the change in the Raft implementation, duplicate data will occur with this upgrade procedure.
- Stop all external publishers to kdb Insights Enterprise
- Wait for all published messages to be available for querying from the database - this will reduce (but not eliminate) the amount of duplicate data
-
Backup any assemblies that may be running
$ kxi assembly backup --filepath assembly-state-file.yaml Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to assembly-state-file.yaml
-
Teardown assemblies named in the output of the backup command
$ kxi assembly teardown --name dfx-assembly --wait --use-kubeconfig Tearing down assembly dfx-assembly Are you sure you want to teardown dfx-assembly [y/N]: y Waiting for assembly to be torn down [------------------------------------] 0% $ kxi assembly teardown --name iot-assembly --wait --use-kubeconfig Tearing down assembly iot-assembly Are you sure you want to teardown iot-assembly [y/N]: y Waiting for assembly to be torn down [------------------------------------] 0%
-
Delete the RT PVCs
kubectl get pvc -n <namespace> | grep sequencer-store | awk '{ print $ 1 }' | xargs kubectl delete pvc -n <namespace>
-
Carry out the upgrade
-
Restore assemblies
kxi assembly deploy --filepath assembly-state-file.yaml --use-kubeconfig
User import
The initUser
will be automatically imported on initial install but not on upgrade. This prevents the initUser
password from being reset on upgrade.
A new value has been added to allow you to change this behavior from the default.
keycloak:
importUsers: false
Default password policy
The default password policy has been updated to include the following rule
- Not one of the previous 24 passwords
If you are upgrading from an earlier version and set importUsers: true
, the password for initUser
must be updated to a new, valid password in the values file before upgrade.
Default Keycloak Credentials
As of 1.3.0 the Keycloak initUser
and initClient
credentials are no longer defaulted within the kdb Insights values.yaml
. See here for more details.
Downgrading to earlier versions
Downgrading to earlier versions was previously supported by executing kxi install upgrade
. This is no longer supported. To install an earlier version, execute the rollback steps .
Schema Configuration Changes
The field updTsCol
is no longer used, and has been removed from the Schema Settings screen. It should be removed from any separately maintained assembly YAML, or configuration made in the Schema Code View screen.
Supported Python versions in the CLI
Support for Python 3.7 has been dropped in the CLI.
Known Issues
-
On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault. -
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes
unable to flush accounting logs
. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.
-
If end-of-day processing is interrupted and remains incomplete until the next EOD period, this prevents further EODs from completing and requires manual recovery.
-
Adding a sort attribute to memory or disk is currently not supported. Changing the attribute to be sorted will not apply the attribute and will error for data on disk.
-
Schema changes will fail when attempting to remove all symbol columns from a schema that previously had any.
-
Historical object storage tiers cannot be added using the UI due to a validation issue. These tiers can still be configured using assembly files submitted using the KXI CLI.
When upgrading, the kdb Insights CLI does not wait until an assembly is fully operative before returning control. There is, therefore, a delay between the kdb Insights CLI returning control and the assembly being ready. If, for any reason, the assembly cannot start, you will not be notified.
-
If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
-
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.
-
When entering a query via the UI, Start and End Date fields are always required. To query against a splayed table or reference data, dummy dates must be entered.
-
When writing Python code in the scratchpad, if you type
from kxi import sp
, it will automatically auto-correct tofrom kxi import super
. The workaround for this issue is as follows:- Accept the auto-correct
- Add a new line to the scratchpad
- Go back up and edit
super
back tosp
- Press the down arrow to go to the next line
- If in the Import Wizard, Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.
The kxi-controller will periodically log errors when an assembly has been deployed from the kdb Insights CLI. These errors can be safely ignored.
"[] Failed to compare saved assembly to deployed version, error: type"
-
When creating a pipeline the user interface does not stop you from creating it with the same name as an existing pipeline, however two pipelines with the same name may not be deploy correctly. Please ensure all pipelines have different names.
-
Any object storage reader nodes in pipelines created in the UI prior to 1.4.0 may need to be edited after the upgrade, as the file path may be split across several path inputs on the properties tab. To remedy, delete the path fields and enter the correct path into the reader.
-
If a reader node is added to a pipeline that invokes
.qsp.read.fromFile
using chunking, then compilation fails.
When defining getData aggregations in a View configuration, a semi-colon separated list must be passed in the form of colAlias;aggFn;col
. Symbols lists are currently not supported.
The default CPU resources for PostgreSQL are too low and result in CPU throttling. The CPU resource can be updated by setting the below in your installation configuration values.yaml
.
keycloak:
postgresql:
primary:
resources:
limits:
cpu: 2000m
requests:
cpu: 50m
1.3.4
Release for kdb Insights Enterprise.
Release Date
2023-03-13
Fixes
[FIX] As of 1.3.3, if the Storage Manager falls behind on EODs for more than a full day, IDB links could be written incorrectly when late data was present. If this issue is encountered, manual data changes are required to recover the database including backup of the stream logs from the date of the failure. Please contact KX Support for help with recovery.
This issue is characterized by a log statement from the Storage Manager as follows: "Unexpected condition: EOXa watermark date YYYY.MM.DD is less than HDB stream date YYYY.MM.DD, setting it to match HDB stream date". The IDB may also start reporting "No such file or directory" on ordinal partitions.
Artifacts
type | location |
---|---|
Infrastructure | kxi-terraform-1.3.1.tgz |
kdb Insights Enterprise | insights-1.3.4.tgz |
Operator | kxi-operator-1.3.1.tgz |
CLI | kxicli-1.3.1-py3-none-any.whl |
ODBC Driver | kodbc 1.3.0 |
Java SDK | java-sdk 1.3.0 |
Batch Ingest Known Issues
- In the unlikely event of two batch ingests being done within an end of interval (EOI) event (for example, a 10 min window) a one hour delay would occur for the subsequent ingest, but there would not be any data loss, just a delay.
- If the Storage Manager (SM) crashes during a batch ingest, the ingest will fail. On SM recovery, it will not try again and must be restarted, but no clean-up is required.
Upgrade Notes
- When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.
Resource Coordinator
-
If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" can be misleading in the timeout message. It should be interpreted as "Timed out before we were able to dequeue".
-
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.
Python Scratchpad for Queries
- If the script consists of multiple statements, the console does not print the output of the last evaluation, and instead it only displays the expression before it is evaluated.
S3 Import Wizard
- If you select the source, S3, and click Next, no error will be thrown, which is invalid. At least one file needs to be present.
Keycloak Config CLI
We recommend that you enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.
This can be enabled by setting:
keycloak:
keycloakConfigCli:
enabled: true
in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.
If you use a shared Keycloak instance, this can be enabled by setting:
keycloak-config-cli:
enabled: true
in your values file.
Keycloak initUser password reset
Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth
key.
Default Password Policy
As of 1.3.0 a default password policy is now being enforced.
The default policy is:
- At least one uppercase letter
- At least one lowercase letter
- At least one symbol
- At least one number
- Minimum length of 14 characters
Information about how the policy can be configured and adjusted can be found here.
If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.
Keycloak initUser
The Keycloak initUser
password must satisfy the policy if it is enabled.
Default Keycloak Credentials
As of 1.3.0 the Keycloak initUser
and initClient
credentials are no longer defaulted within the Insights values.yaml
. These defaults would previously create the demoinsights
user and test-publisher
client on a new deployment of kdb Insights Enterprise.
If you currently set initUser.enabled=true
or initClient.enabled=true
within your own values.yaml
, you may receive the following errors at deploy time:
Keycloak initUser has been enabled
The following fields are required to be set
.Values.keycloak.initUser.name
.Values.keycloak.initUser.auth
Keycloak initClient has been enabled
The following fields are required to be set
.Values.keycloak.initClient.clientId
.Values.keycloak.initClient.clientSecret
If enabling the initUser
you are required to set:
keycloak:
initUser:
enabled: true
name: "initUsername"
auth: "initUserPassword"
Where a Keycloak passwordPolicy has been enabled, initUser.auth
must satisfy the policy requirements.
If enabling the initClient
you are required to set:
keycloak:
initClient:
enabled: true
clientId: "initClientID"
clientSecret: "initClientSecret"
Internal Network LoadBalancers
As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer
.
These annotations restrict access to the LoadBalancers
from outside the cluster.
To disable these annotations and permit access from outside the cluster, you must set:
global:
service:
useInternalLBAnnotations: false
For additional configuration options see here.
Assembly blockSize
Changed
The blockSize
configuration within an assembly spec.tables.<table>.blockSize
has been updated with the following semantics:
- if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
- this has the highest performance, but has no RAM limit on received data
- if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
- the smaller this number is set to, the worse ingest performance but stronger RAM limits; this should be balanced
Previously, blockSize
was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize
field in the assembly for each table.
Known Issues
-
On startup of pods, the following error might be observed after approximately three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault. -
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes
unable to flush accounting logs
. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.
- Setting SM replicas (defining
size
greater than 1) in assembly YAML will cause writedown/storage and query problems. Thesize
parameter for SM should always be set to 1.
sm:
size: 1
- If the cluster and/or resource configuration for the
kxi-discovery-service
is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.
- Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls
'Rollback' (re-apply resources) to the upgraded version:
helm rollback <release name> <current revision>
There are two known issues with the UI logout.
- After performing a logout action, you will not be redirected back to the login screen.
- If you were logged out due to inactivity, you might see a "Logout failed" error.
In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}
.
1.3.3
Release for kdb Insights Enterprise.
Release date
2023-01-17
New features
Functionality for improved ingestion, efficiency and reduction of resources. Take advantage of an IDE to develop your databases, pipelines and query interfaces.
[NEW] Improved ingestion
-
[Beta] Ingest large batches of data directly into the historical database:
- Use fewer resources. Memory and disk resources greatly reduced during ingest.
- Select the
Direct Writing
field in the Database Writer in the UI, or use the q and Python APIs.
See details here for known issues.
[NEW] Writing data to Amazon S3
- You can now write data to Amazon S3 by either:
- Selecting the Amazon S3 Writer node in the UI.
- Using the q and Python APIs.
[NEW] Improved IDE capabilities
-
Build YAML files and upload them using kdb Insights CLI:
- Speed up development workflow, improve productivity.
- Access to the Kubernetes control plane is no longer needed. This is now handled for you by the CLI.
- Assembly management using CLI.
-
If you have access to KX Analyst, you can use it to develop functions. You can now:
- Enjoy a rich IDE development environment with syntax highlighting for YAML files.
- Query kdb Insights Enterprise data using REST API or query interface.
- Upload using the CLI or as part of CI/CD pipeline.
- Build a complete SDLC that harnesses the flexibility of KX Analyst against kdb Insights Enterprise data.
Note
Contact your KX Sales representative if you are interested in using KX Analyst. In future releases, additional IDEs will become available, including VS Code and Jupyter Notebooks.
Improvements
[NEW] Reliable Transport
- Reliable Transport's logging default level for the sequencer and merger has been changed from DEBUG to INFO.
Fixes
[FIX] Following a hard reset of the Reliable Transport (RT), the log files in the out
folder are truncated rather than deleted. This means that the replicated log files for all subscribers to the RT instance will be garbage collected once the RT archiver starts after the reset. This prevents a hard reset from causing subscriber PVCs to fill up due to old/corrupt log files remaining.
Artifacts
type | location |
---|---|
Infrastructure | kxi-terraform-1.3.1.tgz |
kdb Insights Enterprise | insights-1.3.3.tgz |
Operator | kxi-operator-1.3.1.tgz |
CLI | kxicli-1.3.1-py3-none-any.whl |
ODBC Driver | kodbc 1.3.0 |
Java SDK | java-sdk 1.3.0 |
RT Bridge | rt-bridge-1.4.0-install.sh |
Known Issues
Batch ingest
- In the unlikely event of two batch ingests being done within an end of interval (EOI) event (for example, a 10 min window) a one hour delay would occur for the subsequent ingest, but there would not be any data loss, just a delay.
- If the Storage Manager (SM) crashes during a batch ingest, the ingest will fail. On SM recovery, it will not try again and must be restarted, but no clean-up is required.
Discovery Service
- If the cluster and/or resource configuration for the
kxi-discovery-service
is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.
Upgrade Notes
Upgrading assemblies
When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.
Terraform Script
Note the version. GKE cluster cannot be deployed using a 1.3.0 version of the script; a 1.3.* version of the client scripts is needed to use Kubernetes v1.22 on GKE.
Keycloak Config CLI
It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.
This can be enabled by setting:
keycloak:
keycloakConfigCli:
enabled: true
in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.
If you use a shared Keycloak instance, this can be enabled by setting:
keycloak-config-cli:
enabled: true
in your values file.
Keycloak initUser password reset
Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth
key.
Default password policy
As of 1.3.0 a default password policy is now being enforced.
The default policy is:
- At least one uppercase letter
- At least one lowercase letter
- At least one symbol
- At least one number
- Minimum length of 14 characters or greater
Information about how the policy can be configured and adjusted can be found here.
If you are upgrading from an earlier version, and want the default password policy to be applied, the Keycloak config CLI must be enabled.
Keycloak initUser
The Keycloak initUser
password must satisfy the policy if it is enabled.
Default Keycloak Credentials
As of 1.3.0 the Keycloak initUser
and initClient
credentials are no longer defaulted within the kdb Insights values.yaml
. These defaults would previously create the demoinsights
user and test-publisher
client on a new deployment of kdb Insights Enterprise.
If you currently set initUser.enabled=true
or initClient.enabled=true
within your values.yaml
, you may receive the following errors at deploy time:
Keycloak initUser has been enabled
The following fields are required to be set
.Values.keycloak.initUser.name
.Values.keycloak.initUser.auth
Keycloak initClient has been enabled
The following fields are required to be set
.Values.keycloak.initClient.clientId
.Values.keycloak.initClient.clientSecret
If enabling the initUser
you are required to set:
keycloak:
initUser:
enabled: true
name: "initUsername"
auth: "initUserPassword"
Where Keycloak passwordPolicy has been enabled, initUser.auth
must satisfy the policy requirements.
If enabling the initClient
you are required to set:
keycloak:
initClient:
enabled: true
clientId: "initClientID"
clientSecret: "initClientSecret"
Internal Network LoadBalancers
As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer
.
These annotations restrict access to the LoadBalancers
from outside the cluster.
To disable these annotations and permit access from outside the cluster, set:
global:
service:
useInternalLBAnnotations: false
For additional configuration options see here.
Assembly blockSize
changed
The blockSize
configuration within an assembly spec.tables.<table>.blockSize
has been updated with the following semantics:
- If unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval. This has the highest performance, but has no RAM limit on received data
- If set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM. The smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced
Previously, blockSize
was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize
field in the assembly for each table.
Known Issues
-
On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault. -
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes
unable to flush accounting logs
. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.
- Setting SM replicas (defining
size
greater than 1) in assembly YAML will cause writedown/storage and query problems. Thesize
parameter for SM should always be set to 1.
sm:
size: 1
- Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls
'Rollback' (re-apply resources) to the upgraded version:
helm rollback <release name> <current revision>
There are two known issues with the UI logout.
- After performing a logout action, the user will not be redirected back to the login screen.
- In the case a user was logged out due to inactivity, a "Logout failed" error might be observed.
In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}
-
If the Resource Coordinator (RC) times out a request on a dequeue, the displayed message "Unknown reason" should be interpreted as "Timed out before we were able to dequeue".
-
If two processes have the same host and port, then the second one to register will overwrite the first, meaning the first will be invisible to the RC, and consequently never receive any requests.
If the script consists of multiple statements, the console does not print the output of the last evaluation, and instead it only displays the expression before it is evaluated.
If S3 is selected as the source and Next is clicked, no error will be thrown, which is invalid. At least one file needs to be present.
1.3.2
Release for kdb Insights Enterprise.
Release Date
2022-12-05
New Features
[NEW] Pipelines
-
[Beta] Compressed files (GZIP) can now be decoded using the q and Python APIs.
-
Multiple file paths can be added to a cloud storage Reader node using the
+
button in the UI, or via the q and Python APIs. -
Wildcard support using
*
is now available when using a single file path with the Amazon S3 Reader. -
CSV encoding is now available, using a new node in the UI, or via the q and Python APIs.
-
It is now possible to write to Amazon S3 buckets, using a new node in the UI, or via the q and Python APIs.
-
Pipeline lifecycle events can be subscribed to via the q API. This can be used, for example, to invoke a callback once files have been processed by a pipeline.
Improvements
- Queries will no longer time out after 60 seconds at the gateway. Longer timeouts can be configured in
getData
API calls using thetimeout
header option.
Fixes
[FIX] Query
- An issue where DAPs could enter low-memory mode earlier than they should has been resolved.
Artifacts
type | location |
---|---|
Infrastructure | kxi-terraform-1.3.0.tgz |
Platform | insights-1.3.2.tgz |
Operator | kxi-operator-1.3.1.tgz |
CLI | kxicli-1.3.0-py3-none-any.whl |
ODBC Driver | kodbc 1.3.0 |
Java SDK | java-sdk 1.3.0 |
Upgrade notes
Upgrading assemblies
When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.
Keycloak Config CLI
It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.
This can be enabled by setting:
keycloak:
keycloakConfigCli:
enabled: true
in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.
If you use a shared Keycloak instance, this can be enabled by setting:
keycloak-config-cli:
enabled: true
in your values file.
Keycloak initUser password reset
Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth
key.
Default password policy
As of 1.3.0 a default password policy is now being enforced.
The default policy is:
- At least one uppercase letter
- At least one lowercase letter
- At least one symbol
- At least one number
- Minimum length of 14 characters or greater
Information about how the policy can be configured and adjusted can be found here.
If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.
Keycloak initUser
The Keycloak initUser
password must satisfy the policy if it is enabled
Default Keycloak Credentials
As of 1.3.0 the Keycloak initUser
and initClient
credentials are no longer defaulted within kdb Insights Enterprise values.yaml
. These defaults would previously create the demoinsights
user and test-publisher
client on a new deployment of kdb Insights Enterprise.
Users who currently set initUser.enabled=true
or initClient.enabled=true
within their own values.yaml
may receive the following errors at deploy time:
Keycloak initUser has been enabled
The following fields are required to be set
.Values.keycloak.initUser.name
.Values.keycloak.initUser.auth
Keycloak initClient has been enabled
The following fields are required to be set
.Values.keycloak.initClient.clientId
.Values.keycloak.initClient.clientSecret
If enabling the initUser
you are required to set:
keycloak:
initUser:
enabled: true
name: "initUsername"
auth: "initUserPassword"
Where keycloak passwordPolicy has been enabled, initUser.auth
must satisfy the policy requirements.
If enabling the initClient
you are required to set:
keycloak:
initClient:
enabled: true
clientId: "initClientID"
clientSecret: "initClientSecret"
Internal Network LoadBalancers
As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer
.
These annotations restrict access to the LoadBalancers
from outside the cluster.
To disable these annotations and permit access from outside the cluster, the user is required to set:
global:
service:
useInternalLBAnnotations: false
For additional configuration options see here.
Assembly blockSize
changed
The blockSize
configuration within an assembly spec.tables.<table>.blockSize
has been updated with the following semantics:
- if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
- this has the highest performance, but has no RAM limit on received data
- if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
- the smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced
Previously, blockSize
was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize
field in the assembly for each table.
Known Issues
-
On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It is independent from the main processes and does not indicate any application fault. -
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes
unable to flush accounting logs
. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.
- Setting SM replicas (defining
size
greater than 1) in assembly YAML will cause writedown/storage and query problems. Thesize
parameter for SM should always be set to 1.
sm:
size: 1
- If the cluster and/or resource configuration for the
kxi-discovery-service
is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.
- Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls
'Rollback' (re-apply resources) to the upgraded version:
helm rollback <release name> <current revision>
There are two known issues with the UI logout.
- After performing a logout action, the user will not be redirected back to the login screen.
- In the case a user was logged out due to inactivity, a "Logout failed" error might be observed
In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}
1.3.1
Release for kdb Insights Enterprise.
Release Date
2022-11-23
Security
[FIX] kdb Insights Enterprise 1.3.1 contains security updates for a number of critical CVEs
Streams and Reliable Transport
[FIX] An issue which could cause RT archiving to fail has been fixed.
Artifacts
type | location |
---|---|
Infrastructure | kxi-terraform-1.3.0.tgz |
Platform | insights-1.3.1.tgz |
Operator | kxi-operator-1.3.1.tgz |
CLI | kxicli-1.3.0-py3-none-any.whl |
ODBC Driver | kodbc 1.3.0 |
Java SDK | java-sdk 1.3.0 |
Upgrade notes
Upgrading assemblies
When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.
Keycloak Config CLI
It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.
This can be enabled by setting:
keycloak:
keycloakConfigCli:
enabled: true
in your values file if you are deploying Keycloak as a part of kdb Insights Enterprise.
If you use a shared Keycloak instance, this can be enabled by setting:
keycloak-config-cli:
enabled: true
in your values file.
Keycloak initUser password reset
Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth
key.
Default password policy
As of 1.3.0 a default password policy is now being enforced.
The default policy is:
- At least one uppercase letter
- At least one lowercase letter
- At least one symbol
- At least one number
- Minimum length of 14 characters or greater
Information about how the policy can be configured and adjusted can be found here
If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.
Keycloak initUser
The Keycloak initUser
password must satisfy the policy if it is enabled
Default Keycloak Credentials
As of 1.3.0 the Keycloak initUser
and initClient
credentials are no longer defaulted within the kdb Insights Enterprise values.yaml
. These defaults would previously create the demoinsights
user and test-publisher
client on a new deployment of kdb Insights Enterprise.
Users who currently set initUser.enabled=true
or initClient.enabled=true
within their own values.yaml
may receive the following errors at deploy time:
Keycloak initUser has been enabled
The following fields are required to be set
.Values.keycloak.initUser.name
.Values.keycloak.initUser.auth
Keycloak initClient has been enabled
The following fields are required to be set
.Values.keycloak.initClient.clientId
.Values.keycloak.initClient.clientSecret
If enabling the initUser
you are required to set:
keycloak:
initUser:
enabled: true
name: "initUsername"
auth: "initUserPassword"
Where keycloak passwordPolicy has been enabled, initUser.auth
must satisfy the policy requirements.
If enabling the initClient
you are required to set:
keycloak:
initClient:
enabled: true
clientId: "initClientID"
clientSecret: "initClientSecret"
Internal Network LoadBalancers
As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer
.
These annotations restrict access to the LoadBalancers
from outside the cluster.
To disable these annotations and permit access from outside the cluster, the user is required to set:
global:
service:
useInternalLBAnnotations: false
For additional configuration options see here.
Assembly blockSize
changed
The blockSize
configuration within an assembly spec.tables.<table>.blockSize
has been updated with the following semantics:
- if unset: all data received in an interval will be buffered in RAM within SM, written down at the end of the interval
- this has the highest performance, but has no RAM limit on received data
- if set: once a table's rows surpass the configured limit, buffered data will be flushed to disk to release RAM
- the smaller this number is set to, the worse ingest performance but stronger RAM limits - this should be balanced
Previously, blockSize
was ignored, always buffering all data in memory. To reproduce previous behaviour, unset the blockSize
field in the assembly for each table.
Known Issues
-
On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault. -
On initial startup of kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes
unable to flush accounting logs
. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.
- Setting SM replicas (defining
size
greater than 1) in assembly YAML will cause writedown/storage and query problems. Thesize
parameter for SM should always be set to 1.
sm:
size: 1
- If the cluster and/or resource configuration for the
kxi-discovery-service
is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here.
- Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls
'Rollback' (re-apply resources) to the upgraded version:
helm rollback <release name> <current revision>
There are two known issues with the UI logout.
- After performing a logout action, the user will not be redirected back to the login screen.
- In the case a user was logged out due to inactivity, a "Logout failed" error might be observed
In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}
1.3.0
Release for kdb Insights Enterprise.
Release Date
2022-10-26
Features
[NEW] User Interface
- Items in the page tree have been renamed for clarity and consistency with the CLI.
- Data is now known as Assemblies
- Explore is now known as Queries
- Reports is now known as Views
[NEW] Pipelines
-
An "Auto" setting for Parse Strings in the Apply Schema node has been added. The column will be parsed if the input is textual (strings or bytes), but the output type is non-textual.
-
Data types are now visible for each column for Test Deploy previews. These can be used to validate the data types and the data shape of Transforms and Maps results.
-
Data types and column names are validated for Database writers as part of a test deploy.
-
Additional Reader validation has been introduced to pipelines, to ensure correctness of settings before deploying.
-
Test deploy previews can now show non-tabular data.
-
Previews are now available for Reader nodes, before data has been decoded.
-
The visibility of a Pipeline's state has been improved in the Overview tab. Warning and error counts can be clicked on to launch a diagnostics tab for that Pipeline.
[NEW] Assemblies
-
It is now possible to see assemblies created using
kxi assembly create
in the UI. Note that assemblies created and managed through the command-line are not editable via the UI. -
Additional logging has been added to the Storage Manager, to ease troubleshooting.
-
Schema configuration validation has been improved.
[NEW] Install and Packages
-
The kdb Insights Operator is no longer uninstalled by default by
kxi install delete
, the command line parameter--uninstall-operator
must now be used. -
Many values for
kxi install setup
can now be passed via command line parameter, please see the CLI Reference for more detail. -
Improved support for rolling back to previous versions. Enter a previous version number using the upgrade command
kxi install upgrade
to roll back. -
The current
kubectl
context is now used as the default in all commands, instead of the namespace specified in thecli-config
file. -
kdb Insights Python modules for packaging and machine learning are no longer dependent on
libc.musl
, and can now be used for development on Windows and Mac. -
Custom user query APIs and Aggregation functions can be installed and initialised within Data Access Processes and Aggregators as outlined here.
-
Custom user query APIs and Aggregation functions can be installed and initialised within Data Access Processes and Aggregators as outlined here.
[NEW] Observability
- Reliable Transport diagnostics are now available in diagnostics exports from the UI.
[NEW] Infrastructure
- The base image for most components has been moved to Rocky Linux.
Fixes
A large number of fixes have gone into kdb Insights Enterprise 1.3.0. Known issues which have been resolved, along with some other highlights, are listed below.
[FIX] Streams and Reliable Transport
- A hard reset can now be performed on Reliable Transport, to recover from stream corruption. Please contact KX Support for usage instructions.
[FIX] Pipelines
-
Pipelines which have their data cleared when they're torn-down can now be re-deployed with the same name. Previously, this would cause RT sequence number problems, stopping data from flowing.
-
Persistent pop-up errors in the UI after resizing a Map node have been resolved.
-
.qsp.read.fromDatabase
can now be used in pipelines. -
Renaming a pipeline node no longer disconnects its edges.
-
Empty pipelines are no longer deployable.
[FIX] User Interface
- Validation fixes so that date and time fields can be entered without use of the picker.
[FIX] Java SDK
- Occasional authentication failures when querying via Java SDK have been resolved. These would previously be seen more frequently on slower connections.
Artifacts
type | location |
---|---|
Infrastructure | kxi-terraform-1.3.0.tgz |
Platform | insights-1.3.0.tgz |
Operator | kxi-operator-1.3.0.tgz |
CLI | kxicli-1.3.0-py3-none-any.whl |
ODBC Driver | kodbc 1.3.0 |
Java SDK | java-sdk 1.3.0 |
Upgrade notes
Upgrading assemblies
When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.
Keycloak Config CLI
It is recommended to enable the Keycloak Config CLI when upgrading to ensure that any realm changes are imported.
This can be enabled by setting:
keycloak:
keycloakConfigCli:
enabled: true
in your values file if you are deploying Keycloak as a part of KX Insights.
If you use a shared Keycloak instance, this can be enabled by setting:
keycloak-config-cli:
enabled: true
in your values file.
Keycloak initUser password reset
Enabling the Keycloak Config CLI will cause the initUser's password to be reset. It will be reset to the value of the keycloak.initUser.auth
key.
Default password policy
As of 1.3.0 a default password policy is now being enforced.
The default policy is:
- At least one uppercase letter
- At least one lowercase letter
- At least one symbol
- At least one number
- Minimum length of 14 characters or greater
Information about how the policy can be configured and adjusted can be found here.
If you are upgrading from an earlier version, and want the default password policy to be applied, the keycloak config CLI must be enabled.
Keycloak initUser
The Keycloak initUser
password must satisfy the policy if it is enabled
Default Keycloak Credentials
As of 1.3.0 the Keycloak initUser
and initClient
credentials are no longer defaulted within the kdb Insights Enterprise values.yaml
. These defaults would previously create the demoinsights
user and test-publisher
client on a new deployment of kdb Insights Enterprise.
Users who currently set initUser.enabled=true
or initClient.enabled=true
within their own values.yaml
may receive the following errors at deploy time:
Keycloak initUser has been enabled
The following fields are required to be set
.Values.keycloak.initUser.name
.Values.keycloak.initUser.auth
Keycloak initClient has been enabled
The following fields are required to be set
.Values.keycloak.initClient.clientId
.Values.keycloak.initClient.clientSecret
If enabling the initUser
you are required to set:
keycloak:
initUser:
enabled: true
name: "initUsername"
auth: "initUserPassword"
Where keycloak passwordPolicy has been enabled, initUser.auth
must satisfy the policy requirements.
If enabling the initClient
you are required to set:
keycloak:
initClient:
enabled: true
clientId: "initClientID"
clientSecret: "initClientSecret"
Internal Network LoadBalancers
As of 1.3.0 by default annotations are added to Service resources of type LoadBalancer
.
These annotations restrict access to the LoadBalancers
from outside the cluster.
To disable these annotations and permit access from outside the cluster, the user is required to set:
global:
service:
useInternalLBAnnotations: false
For additional configuration options see here.
Known Issues
- If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
{"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
-
The user-defined variables & functions in the Queries tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)
-
The Queries tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to
insights.scratch.*
-
The Queries tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.
-
On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault. -
On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes
unable to flush accounting logs
. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.
- Setting SM replicas (defining
size
greater than 1) in assembly YAML will cause writedown/storage and query problems. Thesize
parameter for SM should always be set to 1.
sm:
size: 1
- If the cluster and/or resource configuration for the
kxi-discovery-service
is limited, a race condition can occur at startup causing the Discovery Service to be in a crash/restart loop. This can be solved by giving the Discovery Service additional CPU and memory resources; full details on setting custom resources can be found here
- Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls
'Rollback' (re-apply resources) to the upgraded version:
helm rollback <release name> <current revision>
There are two known issues with the UI logout.
- After performing a logout action, the user will not be redirected back to the login screen.
- In the case a user was logged out due to inactivity, a "Logout failed" error might be observed
In both of these cases, the logout has been successful and the user can re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}
1.2.1
Release for kdb Insights Enterprise.
Release Date
2022-09-22
Fixes
kdb Insights Enterprise 1.2.1 is a security update for vulnerabilities in the included images for Keycloak/PostgreSQL.
Artifacts
type | location |
---|---|
Infrastructure | kxi-terraform-1.2.0.tgz |
Platform | insights-1.2.1.tgz |
Operator | kxi-operator-1.2.0.tgz |
CLI | kxicli-1.2.0-py3-none-any.whl |
ODBC Driver | kodbc 1.2.0 |
Java SDK | java-sdk 1.2.0 |
keycloak-server | keycloak-server-0.2.1.tgz |
Upgrade notes
Upgrading assemblies
When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.
Version 1.2.1 requires a custom storageClassName definition
The addition of support for packages introduces a new storageClassName
defined as sharedfiles
, this is defined by default in any deployments of kdb Insights Enterprise that rely on the KX provided Terraform scripts for deployment but must be added manually by a user as outlined here.
Alternatively, within the deployment configuration as outlined here, modify the storageClassName
to an alternative supported RWX storage option for example rookcephfs
.
Keycloak 18
The version of Keycloak used within kdb Insights Enterprise 1.2.0 and 1.2.1 has been upgraded from Keycloak 16 to Keycloak 18.
Please follow the Keycloak 18 migration steps to migrate your Keycloak data prior to upgrading kdb Insights Enterprise from versions older than 1.2.0.
Azure Marketplace upgrade
Upgrading from v1.2.0 Marketplace release has a known issue. Keycloak was upgraded in v1.2.1 which contained non backward compatible changes. The PostgreSQL secret format has changed, and existingSecret from values.yaml has an extra nested key. Please make sure values.yaml has the extra auth
nested key, and that the secret containing PostgreSQL passwords, usually called kxi-postgresql
, has dropped the postgresql prefix from the data keys.
Known Issues
- If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
{"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
-
The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)
-
The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to
insights.scratch.*
-
The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.
-
On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault. -
On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes
unable to flush accounting logs
. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.
-
The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.
-
When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.
Sample extract from the application log:
2022-05-24 20:30:48.103035 INFO 1 Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821 INFO 1 Process 41697 terminated
2022-05-24 20:31:48.128196 INFO 1 Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002 INFO 1 Process 41714 terminated
2022-05-24 20:36:12.700467 INFO 1 Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286 INFO 0 New configuration fetched
2022-05-25 05:04:50.092259 WARN 0 Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686 ERROR 0 Config is too old
2022-05-25 05:05:50.046813 INFO 0 Waiting for the driver to sync the data
- Setting SM replicas (defining
size
greater than 1) in assembly YAML will cause writedown/storage and query problems. Thesize
parameter for SM should always be set to 1.
sm:
size: 1
-
Torn-down pipelines that include Database or Stream Writers cannot be directly restarted after having their resources removed. Upon redeploy, data will not be published downstream from the pipeline. A workaround is to rename the pipeline. Resources are removed by checking "Clear Pipeline State" during teardown in the UI, or by directly deleting the Kubernetes PVCs backing the pipeline. If either of these things are done, the pipeline must be renamed before relaunching it.
-
.qsp.read.fromDatabase
cannot be run in a pipeline to query a kdb Insights Enterprise database. The following example code can be used as a workaround to query data (ex. reference data to be used within a pipeline).
onResponse:{[hdr;data]
if[0 <> hdr`ac;
.sp.log.error ("Query failed host=%s, api=%s, code=%n, error=%s";hdr`gw; hdr`api; hdr`ac; hdr`ai)];
publish data
}
.qsp.onStart {
h:hopen hsym `$.spenv.getReleaseName[],"-sg-gateway:5050";
res: neg[h] (`.kxi.getData; (enlist `table)!(enlist `instruments); `onResponse; ()!());
}
.qsp.run .qsp.read.fromCallback[`publish] .qsp.write.toConsole[]
- Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls
'Rollback' (re-apply resources) to the upgraded version:
helm rollback <release name> <current revision>
There are two known issues with the UI logout.
- After performing a logout action, the user will not be redirected back to the login screen.
- In the case a user was logged out due to inactivity, a "Logout failed" error might be observed
In both of these cases, the logout has been successful and the user can navigate re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}
1.2.0
Release for kdb Insights Enterprise.
Release Date
2022-09-02
Features
[NEW] Install and Packages
-
Upgrade support{:target="blank"} in the
kxi
command line tool for easier upgrades. _Please take note of other Upgrade Notes below. -
You must now provide a namespace when deleting installs using
kxi install delete
in the CLI. -
Ability to list, add and delete custom versioned user code via packages to kdb Insights Enterprise via the CLI outlined here.
-
Support within packages for the definition of User Defined Functions (UDFs) written in Python or q for deployment to stream-processor or use in the Explore Window.
[NEW] Pipelines
-
MQTT QoS 0 Reader has been added.
-
Users can run Test deploys in the UI, which provide user feedback on issues when building or editing a pipeline. This includes visualization of the data at each node in the pipeline, and clear indications of which nodes failed.
-
CPU and Memory resources can now be configured from the User Interface to allow rightsizing of data pipelines.
-
Use of User Defined Functions as part of a pipeline.
-
Apply Schema is now available for use after the CSV decoder, so the optional schema parameters for the CSV node have been removed from the Pipelines UI. Users should take care to ensure they're parsing all fields in the Apply Schema node when it's being applied to CSV decoded data.
[NEW] Explore and Query
-
Significant speed improvements to Explore. Query environments are pre-allocated as part of Data/assemblies, and so switching between tabs, assemblies, or instances is now instant. Query environments must be enabled in the Database/Data Access screen, or in assembly configuration, for the assembly to be available in Explore.
-
Filtering and aggregation are now available on the Query tab.
-
There is no need to choose a database when using the Query or SQL options in the Explore window. Label filtering can be used to select particular databases. You can now read from tables with the same name and schema across multiple databases when using the Query or SQL tabs.
-
Import and use of User Defined Functions in pipelines and the Explore Window to allow running of custom code.
-
Console output can now be cleared by right-clicking in the Console and choosing "Clear".
-
The query section at the top of Explore tabs can now be collapsed for a larger code area.
[NEW] Databases
-
CPU and Memory resources can now be configured from the User Interface to allow rightsizing of storage and query services.
-
Redundant replicas possible for all query-path processes, enabling greater query uptime despite process, node, or availability-zone failure.
-
Ability for Data Access tiers (RDB, IDB, HDB) to share compute resources by using
mountList
within Assembly configuration rather thanmountName
. -
Granular assembly deployment status showing the state of deploying components, and assembly deployment errors, are now surfaced in the UI. Mouse-over the assembly's status icon while it's deploying to see the breakdown.
-
Environmental variables can now be configured in the UI for Data Access and Storage when the Advanced view is enabled.
-
Code View for Schemas is now available in the UI. Users can edit (or copy/paste) schemas directly, rather than configuring through UI forms.
[NEW] Performance
- Cluster overprovisioning support that can be configured to enable faster scaling, by reserving resources.
[NEW] Java SDK
-
Query support has been added to the Java SDK.
-
For ingestion there is an option to validate the data being ingested to ensure it is in the format that is expected by the database, before it is ingested.
-
Batch publishing is available, if required.
[NEW] Observability
- It is now possible to filter for a displayed value (for example, a particular Container) in the Diagnostics screen. Right-click on the value and select "Filter by [...]".
[NEW] Infrastructure
- A storage class named
sharedfiles
, that is backed by shared storage, is automatically created when deploying via client Terraform scripts.sharedfiles
storage is used for storing Packages and required by kdb Insights Enterprise.
Fixes
A large number of fixes have gone into kdb Insights Enterprise 1.2.0. Known issues which have been resolved, along with some other highlights, are listed below.
[FIX] Pipelines
- Pipelines no longer report an initial "Unresponsive" status in the UI when launching.
[FIX] Explore and Query
-
With the introduction of Query Environments, there is no longer an issue where PVCs are not removed upon assembly teardown due to active Explore tabs targeting the assembly.
-
SQL queries can now target splayed tables (reference data).
[FIX] Observability
- All UI notifications now appear in the Diagnostics panel.
[FIX] UI
- Redeploying Data (assembly) no longer adds extra nodes to the navigation tree.
Artifacts
type | location |
---|---|
Infrastructure | kxi-terraform-1.2.0.tgz |
Platform | insights-1.2.0.tgz |
Operator | kxi-operator-1.2.0.tgz |
CLI | kxicli-1.2.0-py3-none-any.whl |
ODBC Driver | kodbc 1.2.0 |
Java SDK | java-sdk 1.2.0 |
Upgrade notes
Upgrading assemblies
When upgrading kdb Insights Enterprise, assemblies deployed from the UI in earlier versions will be restarted but will not be automatically upgraded. To upgrade these assemblies, please restart them from the UI post-upgrade.
Version 1.2.0 requires a custom storageClassName definition
The addition of support for packages introduces a new storageClassName
defined as sharedfiles
, this is defined by default in any deployments of kdb Insights Enterprise that rely on the KX provided Terraform scripts for deployment but must be added manually by a user as outlined here.
Alternatively, within the deployment configuration as outlined here, modify the storageClassName
to an alternative supported RWX storage option for example rookcephfs
.
Keycloak 18
The version of Keycloak used within kdb Insights Enterprise has been upgraded from Keycloak 16 to Keycloak 18.
Please follow the Keycloak 18 migration steps to migrate your Keycloak data prior to upgrading kdb Insights Enterprise.
Known Issues
- If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
{"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
-
The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)
-
The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to
insights.scratch.*
-
The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.
-
On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault. -
On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes
unable to flush accounting logs
. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.
-
The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.
-
When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.
Sample extract from the application log:
2022-05-24 20:30:48.103035 INFO 1 Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821 INFO 1 Process 41697 terminated
2022-05-24 20:31:48.128196 INFO 1 Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002 INFO 1 Process 41714 terminated
2022-05-24 20:36:12.700467 INFO 1 Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286 INFO 0 New configuration fetched
2022-05-25 05:04:50.092259 WARN 0 Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686 ERROR 0 Config is too old
2022-05-25 05:05:50.046813 INFO 0 Waiting for the driver to sync the data
- Setting SM replicas (defining
size
greater than 1) in assembly YAML will cause writedown/storage and query problems. Thesize
parameter for SM should always be set to 1.
sm:
size: 1
-
Torn-down pipelines that include Database or Stream Writers cannot be directly restarted after having their resources removed. Upon redeploy, data will not be published downstream from the pipeline. A workaround is to rename the pipeline. Resources are removed by checking "Clear Pipeline State" during teardown in the UI, or by directly deleting the Kubernetes PVCs backing the pipeline. If either of these things are done, the pipeline must be renamed before relaunching it.
-
.qsp.read.fromDatabase
cannot be run in a pipeline to query a kdb Insights Enterprise database. The following example code can be used as a workaround to query data (ex. reference data to be used within a pipeline).
onResponse:{[hdr;data]
if[0 <> hdr`ac;
.sp.log.error ("Query failed host=%s, api=%s, code=%n, error=%s";hdr`gw; hdr`api; hdr`ac; hdr`ai)];
publish data
}
.qsp.onStart {
h:hopen hsym `$.spenv.getReleaseName[],"-sg-gateway:5050";
res: neg[h] (`.kxi.getData; (enlist `table)!(enlist `instruments); `onResponse; ()!());
}
.qsp.run .qsp.read.fromCallback[`publish] .qsp.write.toConsole[]
- Upon upgrade or downgrade, the API Gateway containers may enter a CrashLoopBackOff state. Resources can be reapplied by performing a 'rollback' to the upgraded version. Get the upgraded version by looking at the output from:
helm ls
'Rollback' (re-apply resources) to the upgraded version:
helm rollback <release name> <current revision>
There are two known issues with the UI logout.
- After performing a logout action, the user will not be redirected back to the login screen.
- In the case a user was logged out due to inactivity, a "Logout failed" error might be observed
In both of these cases, the logout has been successful and the user can navigate re-login by navigating back to the main page, i.e. https://${INSIGHTS_HOSTNAME}
1.1.2
kdb Insights Enterprise 1.1.2 is a patch release that corrects a small number of observability and query issues.
Release Date
2022-07-15
Observability
[FIX] Metrics responses are all now terminated with a newline \n
. Absence of the newline caused import issues with some metrics scrapers, for example Azure Monitor.
Explore and Queries
[FIX] The getData API endpoint no longer returns an error when an outputTZ
is provided.
[FIX] Queries against the interval tier (idb) around the end-of-day rollover process could return zero results.
Infrastructure
[NEW] The Terraform scripts will now pass the correct platform to the Docker container when running them on Macs
[FIX] The Helm version in the Terraform scripts' Dockerfile has been pinned to a version without AWS compatibility issues.
Artifacts
type | location |
---|---|
Infrastructure | kxi-terraform-1.1.1.tgz |
Platform | insights-1.1.2.tgz |
Operator | kxi-operator-1.1.0.tgz |
CLI | kxicli-0.9.0-py3-none-any.whl |
ODBC Driver | kodbc 1.1.0 |
Java SDK | java-sdk 1.1.0 |
Upgrade notes
Data loss of UI assembly definitions for versions earlier than 1.1.1
Assembly definitions built from the kdb Insights Enterprise UI in versions earlier than 1.1.1 will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.
To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace.
kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz
To restore, the user should run the following command post-install.
cat kxi-controller-data.tgz | kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -
kdb Insights Operator Version
As the version of the kdb Insights Operator has not changed from 1.1.0 to 1.1.2, the command for deploying kdb Insights Enterprise is slightly different than in earlier releases. The older kdb Insights Operator version must be explicitly passed to the CLI.
kxi install run --filepath values.yaml --version 1.1.2 --operator-version 1.1.0
Please see the guide to Installing kdb Insights Enterprise or Upgrading kdb Insights Enterprise for more detail.
New functionality and roles
kdb Insights Enterprise 1.1.1 includes new features permissioned by additional roles. If you are upgrading to 1.1.2 from a version earlier than 1.1.1, please see the section about authentication upgrades for more information about how to incorporate realm changes, including new roles, during an upgrade.
Known Issues
- If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
{"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
-
The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)
-
Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.
-
Currently users will be logged out but will not be notified via the UI. If users consistently encounter errors, we suggest they log out and in again.
-
The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to
insights.scratch.*
-
The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.
-
The drop-down list of assemblies in the first Explore tab opened after applying the first kdb Insights Enterprise assembly will take a short while to populate. Try again after a few seconds if nothing appears immediately, or close and re-open the Explore tab to see the list and select an assembly.
-
Teardown cleanups of Data (assemblies) via the UI will not fully remove PVCs if there are active Explore tabs targeting that assembly. Close these tabs or remove their underlying sandboxes via the REST API to complete the cleanup.
-
On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault. -
On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes
unable to flush accounting logs
. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.
-
The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.
-
When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.
Sample extract from the application log:
2022-05-24 20:30:48.103035 INFO 1 Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821 INFO 1 Process 41697 terminated
2022-05-24 20:31:48.128196 INFO 1 Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002 INFO 1 Process 41714 terminated
2022-05-24 20:36:12.700467 INFO 1 Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286 INFO 0 New configuration fetched
2022-05-25 05:04:50.092259 WARN 0 Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686 ERROR 0 Config is too old
2022-05-25 05:05:50.046813 INFO 0 Waiting for the driver to sync the data
1.1.1
kdb Insights Enterprise 1.1.1 is a patch release that improves usability and stability.
Release Date
2022-06-28
Data (Assemblies)
[FIX] Clearer tooltips for Data status to ensure users can clearly identify when their assemblies are fully deployed.
[FIX] Additional validation has been added to stop re-deployment of Data (assemblies) after schema changes – schema changes on deployed assemblies are not supported in this version of kdb Insights Enterprise.
[FIX] Additional validation has been added for schemas to ensure they can be deployed successfully.
[FIX] The serviceVolume setting is now mapped correctly in Database and Stream
[FIX] A condition where Stream log archiving could fail has been resolved, ensuring that stream log files do not grow beyond the defined limits.
[FIX] Navigation tree issues where schemas could not be opened and deletion failures were not correctly reflected are both resolved.
Pipelines
[FIX] Nodes requiring passwords no longer store them in the pipeline configuration, they must be entered during pipeline deployment.
[FIX] Mandatory fields in nodes are now enforced in pipelines before deployment.
[FIX] Pipeline re-deployments can now optionally clear state (destroying previously persisted pipeline data, but not data already in the database) to allow edits to be made and the pipeline to be restarted from scratch.
[FIX] The error Cannot read properties of undefined (reading '0')
which could be thrown when opening multiple pipeline tabs has been resolved.
[FIX] An issue where long code lines in Map nodes could cause browsers to hang has been resolved.
Explore and Queries
[FIX] Session errors Worker has been closed
and r.map
are no longer thrown.
[FIX] The error No aggregator available
when changing assemblies in an Explore session is no longer thrown.
[FIX] Reference data joins which do not include the foreign column or where the foreign column has a different name are now possible.
[FIX] An issue where certain queries’ WHERE clause would work via REST but fail via Explore has been resolved.
[FIX] An issue where certain queries using count
could throw the error Error: noupdate: `.Q `pn
has been resolved.
Artifacts
type | location |
---|---|
Infrastructure | kxi-terraform-1.1.0.tgz |
Platform | insights-1.1.1.tgz |
Operator | kxi-operator-1.1.0.tgz |
CLI | kxicli-0.9.0-py3-none-any.whl |
ODBC Driver | kodbc 1.1.0 |
Java SDK | java-sdk 1.1.0 |
Upgrade notes
Data loss of UI assembly definitions
Assembly definitions built from the kdb Insights Enterprise UI will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.
To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace.
kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz
To restore, the user should run the following command post-install.
cat kxi-controller-data.tgz | kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -
kdb Insights Operator Version
As the version of the kdb Insights Operator has not changed from 1.1.0 to 1.1.1, the command for deploying kdb Insights Enterprise is slightly different than in earlier releases. The older kdb Insights Operator version must be explicitly passed to the CLI.
kxi install run --filepath values.yaml --version 1.1.1 --operator-version 1.1.0
Please see the guide to Installing kdb Insights Enterprise or Upgrading kdb Insights Enterprise for more detail.
New functionality and roles
kdb Insights Enterprise 1.1.1 includes new features permissioned by additional roles. Please see the section about authentication upgrades for more information about how to incorporate realm changes, including new roles, during an upgrade.
Known Issues
- If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
{"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
-
The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)
-
Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.
-
Currently users will be logged out but will not be notified via the UI. If users consistently encounter errors, we suggest they log out and in again.
-
The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*
-
The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.
-
The drop-down list of assemblies in the first Explore tab opened after applying the first Insights assembly will take a short while to populate. Try again after a few seconds if nothing appears immediately, or close and re-open the Explore tab to see the list and select an assembly.
-
On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault. -
On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes
unable to flush accounting logs
. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.
-
The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.
-
When either the Java SDK or ODBC driver is being used, if kdb Insights Enterprise Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.
Sample extract from the application log:
2022-05-24 20:30:48.103035 INFO 1 Started replicator to :34.142.40.180:5000, pid 41697
2022-05-24 20:31:48.127821 INFO 1 Process 41697 terminated
2022-05-24 20:31:48.128196 INFO 1 Started replicator to :34.142.40.180:5000, pid 41714
2022-05-24 20:36:12.700002 INFO 1 Process 41714 terminated
2022-05-24 20:36:12.700467 INFO 1 Started replicator to :34.142.40.180:5000, pid 41772
2022-05-25 05:04:50.077286 INFO 0 New configuration fetched
2022-05-25 05:04:50.092259 WARN 0 Config parse error at offset 0: Expected opening bracket
2022-05-25 05:05:50.046686 ERROR 0 Config is too old
2022-05-25 05:05:50.046813 INFO 0 Waiting for the driver to sync the data
Backward Compatibility
Please see the release notes for kdb Insights Enterprise 1.1.0 if you're upgrading from an earlier version, for notes about:
1.1.0
Release for kdb Insights Enterprise.
Release Date
2022-06-07
Features
[NEW] Storage and Query
- Late and out-of-order data handling
- Data received out of order is recorded with an appropriate ingest timestamp
- Late data is recorded, irrespective of where its correct destination is in the lifecycle (RDB, IDB, HDB)
- Late data is included by default for all requests, q (getData) and ANSI SQL
- Late data can be filtered out (for example, to see only data present in the system at a particular point in time)
- Performance is not significantly impacted by late data arrival, including for object storage tiers
- Reference data can be queried in the Explore Window, in a data pipeline and as part of the getData() API
- Support for non time-series reference data in Database
- Data in reference tables can be joined to time-series tables in free-form q queries
- Improved SQL support for queries (joins and complex queries, except for reference tables)
- Compression support, including object storage
- Improved resiliency and performance
[NEW] User Interface
-
Scratchpads
- Data can be pulled into the scratchpad using either sql or getData()
- Freeform q, SQL or Python code can be execute on that data to further filter, aggregate and join data to explore specific records
- Python/q functions or a model can be applied to that data to gain insights
- A model can be fitted using that data
-
Machine Learning plugins can be added to a Pipeline created from the user interface
- More intuitive creation of data ingest pipelines
- Observability embedded into the UI (events and logs tab)
- Log forwarding to each providers' log aggregator is now enabled by default in the Terraform scripts.
[NEW] Reports
- Visualization is available for ingested data using the getData() API
[NEW] Observability
- View Database and Pipeline logs and events from the UI
- Ability to Export Diagnostics, including the logs, events and state for further diagnosis
- Further integration with cloud monitoring tools
- Additional performance metrics collected
[NEW] Reliable Transport Stream
- Subscribers have the option to override the detail truncation options for the stream logs that they have processed
- Improved recovery and performance
Artifacts
type | location |
---|---|
Infrastructure | kxi-terraform-1.1.0.tgz |
Platform | insights-1.1.0.tgz |
Operator | kxi-operator-1.1.0.tgz |
CLI | kxicli-0.9.0-py3-none-any.whl |
ODBC Driver | kodbc 1.1.0 |
Java SDK | java-sdk 1.1.0 |
Upgrade notes
Data loss of UI assembly definition
Data loss of UI assembly definitions
Assembly definitions built from the kdb Insights Enterpirse UI will be lost as part of an uninstall of the application. The assembly components (assemblies, databases, pipelines, reports, schemas, streams) are stored on a PVC in the application. This storage is incorrectly torn down as part of an uninstall, resulting in data loss on re-install or upgrade.
To prevent this data loss the following commands can be run to backup the data. This should be run prior to the uninstall, with kubectl configured to the target cluster and namespace.
kubectl exec insights-kxi-controller-0 -- tar -C /kxic/data -czf - . > kxi-controller-data.tgz
To restore, the user should run the following command post-install.
cat kxi-controller-data.tgz | kubectl exec insights-kxi-controller-0 --stdin -- tar -C /kxic/data -xzf -
Known Issues
- If you request too much data in a single getData(), the request will fail. To overcome this issue you should request less data per getData() call. The error message returned will look like the example below
{"header":{"http":"json","corr":"6dd8f0c5-1895-49c9-a87a-636a834af370","logCorr":"6dd8f0c5-1895-49c9-a87a-636a834af370","client":":10.0.10.76:5050","api":".kxi.getData","protocol":"gw","numRP":1,"ogRcID":"10.0.10.100:5060","to":"2022-04-07T11:41:35.717000000","retryCount":0,"rc":42,"ac":10,"ai":"Agg died"},"payload":[]}
getData
does not support connecting reference data column names that do not match. If the name of a column does not match the name of the column it is a foreign key for, then thegetData
reference join will fail. Example: The trade table has a columnsym
which is a foreign key to themsym
column in the table market. The join will run into an issue since the column names are different.
-
The user-defined variables & functions in the Explore tab are shared across users. (We recommend that you prefix all your variables with characters that are unique to your user.)
-
Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.
-
Passwords used for authenticating to third-party services are currently stored as part of the pipeline configuration. This will be corrected in the next release. It is recommended that instead of using a password directly that a Kubernetes secret is used whenever possible.
-
Currently users will be logged out but will not be notified via the UI. If users consistently encounter errors, we suggest they log out and in again.
-
The Explore tab allows users to execute arbitrary code within the cluster. To lock down the environment, this can be restricted with role based authentication using Keycloak. Restrict the roles relating to insights.scratch.*
-
The Explore tab will not print results to the console for multi-line python code. Results are only printed if there is just one line of python code.
-
On startup of pods, the following error might be observed once roughly after three minutes of a pod starting up
no acct for 3x period, exiting
. This stems from a temporary startup job not shutting down correctly. It's independent from the main processes and doesn't indicate any application fault. -
On initial startup of the kdb Insights Enterprise, there may be some noise printed in the logs while the system initializes
unable to flush accounting logs
. This relates to the capturing of consumption-based license logs and is thrown while all pods get into a running state. It does not indicate any fault in the application and all data should be flushed correctly after a short period.
-
The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.
-
When either the Java SDK or ODBC driver is being used, if kdb Insights Information Service fails to respond to the SDK for at least 1 minute it will deem their own configuration too old and will disconnect and stop publisher into kdb Insights Enterprise. This can be observed when the control plane is inaccessible (for example during the default GCP maintenance window, 5am UTC, if it hasn’t been disabled). Any application using the SDKs will need to account for this and introduce some restart logic which will trigger the SDK to restart, and force a reconnect attempt by the SDK.
Sample extract from the application log:
``` 2022-05-24 20:30:48.103035 INFO 1 Started replicator to :34.142.40.180:5000, pid 41697 2022-05-24 20:31:48.127821 INFO 1 Process 41697 terminated 2022-05-24 20:31:48.128196 INFO 1 Started replicator to :34.142.40.180:5000, pid 41714 2022-05-24 20:36:12.700002 INFO 1 Process 41714 terminated 2022-05-24 20:36:12.700467 INFO 1 Started replicator to :34.142.40.180:5000, pid 41772
e set for the deployment. 2022-05-25 05:04:50.077286 INFO 0 New configuration fetched 2022-05-25 05:04:50.092259 WARN 0 Config parse error at offset 0: Expected opening bracket 2022-05-25 05:05:50.046686 ERROR 0 Config is too old 2022-05-25 05:05:50.046813 INFO 0 Waiting for the driver to sync the data ```
Backward Compatibility
- If you're upgrading from a previous version or the CRD is already installed, helm will not update the definition automatically. See Operator Overview Helm Warning for details on how to rectify this.
License changes
From 1.1.0
kdb Insights has moved to consumption-based licensing using a new license file, kx.lic
.
The user should generate the license themselves using the klic self-service tool. The documentation for this
is available here.
After following the klic docs, a new kx.lic
should be available for your cluster.
Then you can upload it as a Kubernetes secret using the following command.
kubectl create secret generic kxi-license --from-file=license=kx.lic
Alternatively delete your existing license secret and re-run the CLI install (kxi install ..
),
choosing the N
when prompted if you have an existing license secret. Assuming the license
secret is called kxi-license
, the command below will delete it.
kubectl delete secret kxi-license
To revert to the pre-1.1.0 behaviour, the following Helm values should be set for the deployment. For details on this file see here.
global:
license:
..
onDemand: true
asFile: false
kxi-acc-svc:
enabled: false
Single pod Storage Manager
By default, when upgrading to 1.1.0 the Storage Manager will no longer create independent containers for the various processes used to manage data persistence. These processes now all run under a single container as child processes. As part of the upgrade, it is optionally encouraged that any assembly data workflows deployed should be upgraded to align any resources
with this change.
The change requires the user to remove resources
configuration for eoi
, eod
and dbm
sections of an assembly and update the sm.k8sPolicy
with a resources
configuration for all these processes which will run as a single container within the sm
pod.
For example
sm:
..
k8sPolicy:
resources:
requests:
memory: "14Gi"
cpu: "2000m"
limits:
memory: "14Gi"
cpu: "4000m"
However if no actions are taken, kdb Insights will remain backwards compatible with the existing configuration and will use the sum of all CPU and memory resources
under the sm
element, using this sum to populate the sm.k8sPolicy.resources
section under the sm
element.
1.0.0
Release for kdb Insights Enterprise.
Release Date
2022-03-14
Features
[NEW] UI allows storage and reload of data assemblies.
[NEW] Schemas can be created and deleted in the kdb Insights Enterprise user interface.
[NEW] Assemblies can be created, saved and deployed in the kdb Insights Enterprise user interface.
[NEW] Ability to Export Assembly Definitions.
[NEW] Integration with the new License Management system. Due to these changes, existing licenses will need to be re-issued. Please contact your KX representative for more information.
Artifacts
type | location |
---|---|
Infrastructure | kxi-terraform-1.0.0.tgz |
Platform | insights-1.0.0.tgz |
CLI | kxicli-0.4.0-py3-none-any.whl |
Operator | kxi-operator |
ODBC Driver | kodbc 1.0.0 |
Java SDK | java-sdk 1.0.0 |
Known Issues
-
The Service Gateway may repeatably print a
NullPointerException
under certain conditions. If this should occur the Kubernetes pod that the service gateway process is running on should be bounced. -
An OOM event may be triggered when repeatedly bombarded with requests that return a large result set. In the event where an OOM occurs the client will not receive a response; the request may be retried and serviced by other gateways.
-
When choosing an assembly in the Explore Window a ‘500 Internal Server Error’ can be returned if no labels are included in the labels section of the assembly definition.
-
In the prospective menu (the customizable navigation toolbar) not all options available in the menu selector are currently supported, we recommend keeping the default menu options. However, all options are included in the Entity Tree shown when the Home menu is selected.
-
When exporting an Assembly definition from the kdb Insights Enterprise UI the file requires some cleaning before it can be imported into another environment.
- Remove id entry from tables.
- Remove apiVersion entry from pipelines and sequencers.
- Remove line breaks from pipeline specs.
-
When saving a pipeline in the UI on AWS or Azure you need to change the Storage Class from “standard” (which is the default for GCP) to a storage class which exists on AWS or Azure. For example, to “gp2" for AWS, or "default" for Azure. This is done in the Persistence tab when saving. Note, the Storage Class is set separately for the controller and worker. Both must be set correctly. Scroll down to set the worker Storage Class too. Otherwise, the pipeline pod and pvc deployment will get stuck in the “pending” state as no “standard” storage class can be found. Alternatively, create a “standard” storage class per the link storage class resource.
-
Assemblies applied directly through kubectl cannot be edited in the User Interface. However, they can be queried via the Explore tab.
-
Not all stream log files written by the Reliable Transport mechanism (RT) are deleted once the stream has been replicated to the subscriber as the RT stream log archival rules are not being applied to all processes. This will cause the disk space of certain volumes associated with the kdb Insights Enterprise cluster to fill up over time. The more data that is sent through the system the faster this will occur. This may result in out of disk space issues at the producer/subscriber and in the raft cluster. We recommend that you monitor the disk usage and manually delete old stream logs regularly to prevent the disks from filling up. A tool that monitors the disk space could be used to alert you when this intervention is required (for example: Google's Cloud Watch on GCP, or Prometheus). See the section RT Logs for locations of the stream log files and how to delete old stream logs that are not longer required.
-
If a publisher is running and either the out directory on any of your 3 nodes in the RT cluster stops growing, or your Stream Processor is not receiving data, you should restart your publisher. To investigate the progress of messages in the RT cluster stream logs and look at the RT application logs see the section RT Logs.
- If a user leaves trailing spaces after CLI fields, such as secrets, then the trailing spaces are preserved in the values.yaml file leading to errors on deployment. For example,
kxi-certificate
in below yaml:
global:
caIssuer:
name: 'kxi-certificate '
secretName: 'kxi-certificate '
- The CLI error handling is currently limited. If necessary use the underlying kubectl command to see the issue more clearly. For example, to see a yaml syntax error returned by a modified sdk_sample_assembly.yaml file, run:
kubectl apply -f sdk_sample_assembly.yaml
.
-
The KX ODBC driver is not currently available for use on Windows, therefore as the Java SDK requires the ODBC driver to be installed, neither SDKs are currently available on Windows.
-
When using the Sample Java program, provided as part of the Java SDK, replace valFloat:float with valFloat:double in the first line of the sample.csv file before running the program.
- The Controller expects you to call your idb,
myidb
, under instances if you set mounts to use different keynames than your instances an error occurs.
dap:
instances:
idb:
size: 1
mountName: myidb
gwAssembly: insights
- Enabling .z handler metrics accumulates memory over time and may cause containers to restart if they hit their memory resource limit.
Backward Compatibility
- If you're upgrading from a previous version or the CRD is already installed, helm will not update the definition automatically. See Operator Overview Helm Warning for details on how to rectify this.
0.9.0
Beta 1 Release for kdb Insights Enterprise.
Release Date
2021-12-09
Features
[NEW] Run Python and q Machine Learning Models.
-
Users can add a q or Python based model to the ML registry.
-
Users can use q to retrieve either a q or a Python based model from the ML registry.
-
Users can deploy a Stream Processor pipeline that includes ML models from the ML Registry.
[NEW] Run Python functions on streaming data. Users can add Python functions into a Stream Processor pipeline to apply transformations (for example: windowing and filtering) to the streaming data.
[NEW] Support for the Kafka Schema Registry (dynamic schema updates are not yet supported).
[NEW] Users have access a single OpenAPI document. This provides documentation for the following endpoints:
-
Service Gateway
-
Client Controller
-
Information Service
-
Stream Processor
All available operations and authentication methods are documented.
[NEW] The User interface supports using ANSI SQL queries on a single database. These queries are made available via a sandbox to ensure the live database is not modified by the queries.
[NEW] Command line interface for:
-
Installation setup.
-
Enrolling and removing clients.
-
Creating and deleting assemblies.
-
Querying data.
[NEW] Login screens include KX themes
Artifacts
Known Issues
-
Creating a pipeline in the UI that contains an Expression node must be done through the Pipeline editor, the Import screen does not allow correct configuration of an Expression node.
-
If one of the nodes in an RT sequencer is killed or restarts, data may no longer flow through that RT. If the issue occurs on north and south sequencer, it may be sufficient to restart the assembly. Otherwise, the cluster needs to be restarted.
-
RT nodes can under certain conditions consume excessive CPU causing the cluster CPU go up to >95%. If the issue occurs open a shell to the affected node(s) and kill all the pull_server processes. They will be automatically restarted and the CPU usage will drop.
-
RT occasionally reports an inotify error. This is a resource exhaustion issue. If the issue occurs on north and south sequencer, it may be sufficient to restart the assembly. Otherwise, the cluster needs to be restarted.
-
Both SQL and q queries work with assemblies. However, when running a query against a html5evalcongroup process in the UI, SQL is not supported and q must be used instead.
-
Programmatic OpenAPI kdb Insights client generation currently fails due to an issue with the Stream Processor.
Backward Compatibility
- If you're upgrading from a previous version or the CRD is already installed, helm will not update the definition automatically. See Operator Overview Helm Warning for details on how to rectify this.
0.8.0
Initial Alpha Release for kdb Insights Enterprise.
Release Date
2021-10-27
Features
[NEW] Secure login & authentication using Keycloak, enabling enrollment and removal of client publishers.
[NEW] Role Based Access Control with roles for pipeline viewing and editing, access to the user interface and the ability to query data.
[NEW] Licensing is via a base install key.
[NEW] Data ingestion via:
-
Publishing from the RT command line interface.
-
Reading from Kafka.
-
Publishing using direct qIPC.
-
Batch ingestion from S3 buckets.
[NEW] The User interface supports creation of stream processor pipelines; including data ingestion and transformation, and presentation in grids/charts. For this release, pipelines can be published, but drafts are only saved to the user’s browser - not to kdb Insights Enterprise.
[NEW] Data transformations can use q
[NEW] Data transformations can use a limited Python API (built on embedPy)
[NEW] Data can be streamed out via Kafka.
[NEW] Data can be visualized using the kdb Insights user interface
[NEW] Data stored in the kdb databases can be accessed via:
-
The user interface using q
-
A REST API
[NEW] Application Metrics can be generated which can be monitored via Prometheus and Grafana.
Artifacts
type | location |
---|---|
infrastructure | https://nexus.dl.kx.com/repository/kx-insights-packages/kxi-terraform/0.1.7/kxi-terraform-0.1.7.tgz |
installation | https://nexus.dl.kx.com/repository/kx-insights-charts/insights-0.8.0.tgz |
Known Issues
-
On AWS - deployment of infrastructure needs to be targeted at a region that has 3 availability zones - see https://aws.amazon.com/about-aws/global-infrastructure/regions_az/
-
On AWS(us-east-2) - Due to an issue specific to us-east-2 terraform tries to fetch ec2 information by hitting ec2.us-east-2.amazonaws.com normally this url should return a single public ip address but within an ec2 instance on us-east-2 region it returns 3 private ips which are not accessible via https. For this reason the application should not be deployed to us-east-2.
0.10.0
Beta 2 Release for kdb Insights Enterprise.
Release Date
2022-02-04
Features
[NEW] Support for the Kafka Schema Registry has been extended to include dynamic schema updates.
[NEW] Defining ML models can be done using Python.
-
Users can use Python to retrieve either a q or a Python based model from the ML registry.
-
Users can use Python to save either a q or a Python based model from the ML registry.
[NEW] The ANSI SQL support has been extended to include distributed databases. Currently joins are not supported and tables with the same names in different databases MUST have the same columns and properties. As in the previous release these queries are made available via a sandbox to ensure the live database is not modified by the queries.
[NEW] REST API getData function has been enhanced to include:
- Basic aggregation functions on individual columns (for example: min, max, avg, stdev)
- Racking and Filling
- Filtering
- Time range options, including temporaility
[NEW] Schemas, Streams and Databases can be created and deleted in the kdb Insights Enterprise user interface.
[NEW] KX ODBC driver for data ingestion. Enabling data ingestion by applications that support ODBC.
[NEW] KX Java SDK for data ingestion.
Artifacts
type | location |
---|---|
infrastructure | kxi-terraform-0.1.12.tgz |
installation | insights-0.10.0.tgz |
cli | kxicli-0.2.0-py3-none-any.whl |
operator | kxi-operator-0.2.0-rc.19.tgz |
ODBC driver RPM package | kodbc-0.1.52-rhel.x86_64.rpm |
ODBC driver Debian | kodbc-0.1.52.deb |
ODBC driver Windows executable | kodbc-0.1.52.exe |
ODBC driver Windows msi | kodbc-0.1.52.msi |
ODBC driver sample program | kodbc-0.1.52.zip |
Java SDK | rt_client-1.0.0rc6.jar |
Java SDK sample program | rtdemo-1.0.0rc6-all.jar |
Java SDK sample csv file | sample.csv |
Known Issues
-
The sample assembly yaml files provided specify the same subTopic: "data". To start multiple assemblies in parallel, change the subTopic to a unique value, such as subTopic: "odbc-assembly". The corresponding change will also need to be made when enrolling the client. For example: curl https://${INSIGHTS_HOSTNAME}/clientcontroller/enrol -H "Authorization: Bearer
" -d '{ "name": "my-client", "topics": { "insert": "odbc-assembly", "query": "requests" }}' -
The UI may produce pop-up error messages in various scenarios.
-
The UI does not have Reports available.
-
kdb Insights CLI error handling is currently basic. If the CLI throws a stack trace, use the underlying kubectl command to see the issue more clearly. For example, to see a yaml syntax error returned