Changes in 4.5.0
4.5.0 supports kdb+ 4.0.
There are areas in kdb+ 4.0 which are not upwardly compatible.
Red Hat Enterprise 8 support
From V4.5.0, Kx Platform supports the Red Hat Enterprise Linux 8 operating system.
The MS processes are now configured in a cluster by default. This ensures there can only be one master process under normal conditions. This prevents clients registering with the wrong server and not matching topics.
Fixed issued related to the
DxCache group of analytics where incorrect results could be returned in certain circumstances when the cache filled.
Protection against log forging has been added to all Java components.
Any existing log-scraping configuration should be reviewed in case it needs updating to match changes arising from encoded log statements.
deltacomponent user can now be configured as a non-admin. This provides further security by restricting the operations this user can perform on the system. This is set by adding the following to
deltacomponent user to be a non-admin user will have consequences for
- Gateway queries
- Calls to Kx Control or between processes
- String/lambda queries
- Use of
Secured clean installs
Upgrades to existing systems will be unaffected.
Clean installs of Kx Control will be secured by default. The secure parser will be enabled and
deltacomponent set to non-admin.
Upgrades to existing systems will be unaffected.
To override this and maintain the old behavior the following can be included in
install.config before install, or updated in
delta.profile after install and before starting Kx Control
DELTACONTROL_SECURE_PARSER=NO DELTACONTROL_LAMBDA_PERMISSIONED=NO DELTACONTROL_DELTACOMPONENT_ADMIN=YES
Force Control startup
If a process in the Control process cluster joins late and has a higher replication UID, by default it will shut down to trigger manual investigation. This behavior has been made configurable to let it become the master and demote the rest of the cluster.
This scenario can occur if the cluster is not started up all at once, and a master is elected with a lower UID. There will have been data loss in this case so this allows that process to force startup and cause the rest to re-sync to it. The following setting should be set in
Significant work was undertaken around QR performance. The headline changes are given below;
- Improvements of ~30% when processing incoming requests
- New faster FIFO mode
- De-queue performance improvement of ~70% in FIFO
More detailed information on the test scenarios and improvements are available.
The ability to split the QR framework and underlying datasources into shards has been added. This allows for greater request scalability and better flexibility when customizing the framework processes.
The default state of the QR framework is now more secure.
- Permissions enabled on QR and QP instances
- Custom instances will need to make the same change manually
- Password handler enabled on QR process
- Secure parser and async permissions enabled
To revert to the previous behavior, please set the
.qr.pm.secure instance parameter to
Custom process templates
If using custom process templates, it is possible the processes may experience issues connecting to the QR process when trying to register as a database or client. In this case, please add the template name to the following file and restart the process
qp optional parameter within the Query Router can now support providing a list of Query Processors. This will indicate to the Query Router which QPs should be considered for servicing the query with.
QR Garbage Collection configuration
.qr.runGC added to Instance configuration to support garbage collection in the Query Router
Write permission now required for Analyst use on permissioned processes
Write permission is now required for an instance when attaching, if that instance has been configured with permissions enabled.
Permissions for creating sandbox processes
Ability for users to launch
ANALYST_ sandbox process may now be controlled by granting read permission to the CxSandbox entity group.
This group does not exist by default and must be created if desired.
Restrict sandbox creation by daemon, port range, and user group
A Parameter entity titled
CxDaemonAccess may now be used to restrict
Analyst_ sandbox user access to particular daemons, within particular port ranges.
This parameter does not exist by default and must be created if desired.
Kx Control supports dynamic code and schema upgrades to the system. Process API allow an operating version to be set for a process, with respect to the version number of Kx Control. This will ensure config, analytics, instructions and schema will only load the latest version of the entity up to the set version.
Packaging Public API
Kx Platform Process API extended to include packaging commands
- Import packages
.pkg.rollbackpackages and entity revisions (
deltacomponentto be an Administrator)
- View Control
- Interrogate for modified entities
- Republish config updates
Command line interface to Kx Control
There is an option for
run_command.q to suppress logging of the command to
To connect to Kx Control and run a command from a specific list of allowed commands, for example to run a list of workflows, the
control_cli.q script has been provided. The script is invoked with different options to run each of the allowed commands.
Kx Control Web UI
Git compare key change
From 4.5.0 Git functionality within the Kx Control UI will compare Kx Control in memory entities against Git. Previously this compared in-memory entities against on-disk.
Overwrite existing Git repository on import
When importing a new Git repository there may be an existing Git repository that already exists, say if the package was all deleted. If this is the case a warning dialog will be displayed showing the option to cancel the clone or to continue and completely overwrite the existing directory on the file system.
Additional file types in Git commit dialog
The configuration files will now be visible within the Git commit dialog. They are automatically staged and cannot be unstaged since they may be required by changes being pushed.
Git commit a single item from the navigator
The navigator tree, or other lists where entities are displayed, now have a shortcut to Git committing that single item. If the menu item is greyed out, then the item is not new or has not been changed.
All other unrelated entities are moved to the unstaged area of the Git commit dialog. The option to stage the other items is still possible.
Git dialog now offers tooltips for names which extend beyond the size of the dialog.
Search item from entity tab
The Control Web UI tabs have a new menu item Search Item in Navigator in addition to Copy name. This menu item populates the search field in the Navigator.
Duplicate entity now opens editor
When an entity is duplicated in the Control Web UI, an editor is now opened on the newly created entity. Previously, it was required to find the new entity in the navigator and open it manually.
Config Parameter Editor Export option
New option in Config Parameter editor to allow the user to export a parameter to CSV
Schema column search
Added ability to filter/search for information in the schema editor
Several performance improvements have been made to the API populating different areas of Kx Control Web UI. For an environment with large numbers of entities and several hundred packages on disk, example timing improvements are:
|area||time before changes (ms)||time after changes (ms)|
|Process status viewer||20387||58|
|Exporting to GIT (exportStatus)||21135||31|
|Deleting an analytic||20389||7|
The tickerplant has been enhanced to be more resilient to disk outages. This mode is available by enabling the
retryInterval instance parameter.
Messaging per-channel publishes
The publish APIs support publishing to all or specific channels. The per-channel publish APIs,
.dm.pub*c versions take an extra parameter and only publish to subscribers on that channel. In the example below, the second line will only publish to a subset of the subscribers whereas the first will publish to all.
.dm.pub[`trade; data] .dm.pubc[`channel; `trade; data]
There is a quirk in this behavior around the null channel
`. Null is a valid channel in its own right, however these APIs treat it as a wildcard for all channels. In the example above when using the null channel, both lines would be equivalent.
The more intuitive behavior would be for
.dm.pubc to filter on the null channel. To switch to use this behavior, set the following environment variable in your
It is now possible to configure the
deltacomponent user to be a non-administrator user in Kx Control. If this is the case there may be actions required to grant
deltacomponent permission to any solution reports. The administrator status of the
deltacomponent user will not change if an existing environment is upgraded to 4.5.0.
RDB and HDB end of day with secure parser,
deltacomponent user is configued to be a non-administrator user in Kx Control and the secure parser is enabled, there may be issues running end-of-day actions on the HDB from the RDB. This could occur if these actions are defined using string or lambda queries. For an example of how to resolve this see the Kx Control documentation.
Kx Monitoring Dashboards queries now Analytics
Kx Monitoring Dashboards have moved all string based queries to offical Analytic APIs. This ensures the APIs are permissioned appropriately and available via secure deploys.
Kx Java Stream API
Extended the configuration of the Java Stream API to support a variety of additional configuration options, available via command-line
-D system properties or programmably within the application using the Kx Java Stream API.
This includes - Heartbeating and reconnect configurations of Query Router, Query Manager and Messaging Services - Ability to execute queries against only connected Query Processors - Define Messaging Server queue mode and timeouts
Heartbeat log info messages visibility within Java Stream API
Added ability to refresh the client facing public API. This option allows applications to programmatically update the API.
Remote subscribers to the the Kx Java Stream API will now recover when connection is severed to the process on a retry timer configured through the system property
Kx C# Stream API
Kx C# Stream API now provides an interface to allow an application to request the data be presented as a raw byte stream from the backend KDB+ process. This allows the application to deserialize later/off main thread.
Kx C# Stream API adds support for certificate authentication and basic instructions on how to import and trust certificates.
Python script update
The provided example Python script within Kx Connect updated to support both Python 2 and Python 3.
Configurable decimal rounding available
New environmental variable
JSON_DECIMAL_FORMAT available to allow the user to configure the JSON format of decimals.