Changes in 4.5.0

Key changes

Kdb+ 4.0

4.5.0 supports kdb+ 4.0.

Release notes for V4.0

There are areas in kdb+ 4.0 which are not upwardly compatible.

Red Hat Enterprise 8 support

From V4.5.0, Kx Platform supports the Red Hat Enterprise Linux 8 operating system.

Messaging failover

The MS processes are now configured in a cluster by default. This ensures there can only be one master process under normal conditions. This prevents clients registering with the wrong server and not matching topics.

Cache analytics

Fixed issued related to the DxCache group of analytics where incorrect results could be returned in certain circumstances when the cache filled.

Log-forging protection

Protection against log forging has been added to all Java components.

Log scraping

Any existing log-scraping configuration should be reviewed in case it needs updating to match changes arising from encoded log statements.

Kx Control

Secured deltacomponent user

The deltacomponent user can now be configured as a non-admin. This provides further security by restricting the operations this user can perform on the system. This is set by adding the following to delta.profile

DELTACONTROL_DELTACOMPONENT_ADMIN=NO

Setting the deltacomponent user to be a non-admin user will have consequences for

  • Gateway queries
  • Reports,
  • Calls to Kx Control or between processes
  • String/lambda queries
  • Use of .ex.prh/.ex.prhAsync

Security and the deltacomponent user

Secured clean installs

Upgrades to existing systems will be unaffected.

Clean installs of Kx Control will be secured by default. The secure parser will be enabled and deltacomponent set to non-admin.

Upgrades to existing systems will be unaffected.

To override this and maintain the old behavior the following can be included in install.config before install, or updated in delta.profile after install and before starting Kx Control

DELTACONTROL_SECURE_PARSER=NO
DELTACONTROL_LAMBDA_PERMISSIONED=NO
DELTACONTROL_DELTACOMPONENT_ADMIN=YES

New install security

Force Control startup

If a process in the Control process cluster joins late and has a higher replication UID, by default it will shut down to trigger manual investigation. This behavior has been made configurable to let it become the master and demote the rest of the cluster.

This scenario can occur if the cluster is not started up all at once, and a master is elected with a lower UID. There will have been data loss in this case so this allows that process to force startup and cause the rest to re-sync to it. The following setting should be set in delta.profile.

DELTACONTROL_SLAVE_STARTUPOVERWRITE=YES

Replication

Query Router

Performance

Significant work was undertaken around QR performance. The headline changes are given below;

  • Improvements of ~30% when processing incoming requests
  • New faster FIFO mode
  • De-queue performance improvement of ~70% in FIFO

More detailed information on the test scenarios and improvements are available.

QR performance

Sharding

The ability to split the QR framework and underlying datasources into shards has been added. This allows for greater request scalability and better flexibility when customizing the framework processes.

QR Sharding

QR security

The default state of the QR framework is now more secure.

  • Permissions enabled on QR and QP instances
  • Custom instances will need to make the same change manually
  • Password handler enabled on QR process
  • Secure parser and async permissions enabled

To revert to the previous behavior, please set the .qr.pm.secure instance parameter to false.

Screenshot

Instance parameters

Custom process templates

If using custom process templates, it is possible the processes may experience issues connecting to the QR process when trying to register as a database or client. In this case, please add the template name to the following file and restart the process

${DELTACONTROL_HOME}/api/functions.csv

Optional parameters

The qp optional parameter within the Query Router can now support providing a list of Query Processors. This will indicate to the Query Router which QPs should be considered for servicing the query with.

QR Send Request

QR Garbage Collection configuration

Additional configuration .qr.runGC added to Instance configuration to support garbage collection in the Query Router

Enhanced Configuration

Write permission now required for Analyst use on permissioned processes

Write permission is now required for an instance when attaching, if that instance has been configured with permissions enabled.

Permissions for creating sandbox processes

Ability for users to launch ANALYST_ sandbox process may now be controlled by granting read permission to the CxSandbox entity group. This group does not exist by default and must be created if desired.

Analyst sandbox permissions

Restrict sandbox creation by daemon, port range, and user group

A Parameter entity titled CxDaemonAccess may now be used to restrict Analyst_ sandbox user access to particular daemons, within particular port ranges.

This parameter does not exist by default and must be created if desired.

Analyst sandbox restriction

Selective upgrades

Kx Control supports dynamic code and schema upgrades to the system. Process API allow an operating version to be set for a process, with respect to the version number of Kx Control. This will ensure config, analytics, instructions and schema will only load the latest version of the entity up to the set version.

Selective Upgrades

Packaging Public API

Kx Platform Process API extended to include packaging commands

  • Import packages .pkg.import, publish .pkg.release and rollback .pkg.rollback packages and entity revisions (.pkg.import and .pkg.rollback require deltacomponent to be an Administrator)
  • View Control .pkg.getCheckpoints and Package .pkg.getPkgVersion versions
  • Interrogate for modified entities .pkg.getModifiedEntities
  • Republish config updates .cfg.republishUpdates

Packaging commands

Command line interface to Kx Control

There is an option for run_command.q to suppress logging of the command to DeltaControl.log.

To connect to Kx Control and run a command from a specific list of allowed commands, for example to run a list of workflows, the control_cli.q script has been provided. The script is invoked with different options to run each of the allowed commands.

Command-line interface

Kx Control Web UI

Git compare key change

From 4.5.0 Git functionality within the Kx Control UI will compare Kx Control in memory entities against Git. Previously this compared in-memory entities against on-disk.

Overwrite existing Git repository on import

When importing a new Git repository there may be an existing Git repository that already exists, say if the package was all deleted. If this is the case a warning dialog will be displayed showing the option to cancel the clone or to continue and completely overwrite the existing directory on the file system.

Screenshot

Additional file types in Git commit dialog

The configuration files will now be visible within the Git commit dialog. They are automatically staged and cannot be unstaged since they may be required by changes being pushed.

Screenshot

Git commit a single item from the navigator

The navigator tree, or other lists where entities are displayed, now have a shortcut to Git committing that single item. If the menu item is greyed out, then the item is not new or has not been changed.

Screenshot

All other unrelated entities are moved to the unstaged area of the Git commit dialog. The option to stage the other items is still possible.

Screenshot

Git dialog now offers tooltips for names which extend beyond the size of the dialog.

Search item from entity tab

The Control Web UI tabs have a new menu item Search Item in Navigator in addition to Copy name. This menu item populates the search field in the Navigator.

Screenshot

Screenshot

Duplicate entity now opens editor

When an entity is duplicated in the Control Web UI, an editor is now opened on the newly created entity. Previously, it was required to find the new entity in the navigator and open it manually.

Config Parameter Editor Export option

New option in Config Parameter editor to allow the user to export a parameter to CSV

Added ability to filter/search for information in the schema editor

Performance improvements

Several performance improvements have been made to the API populating different areas of Kx Control Web UI. For an environment with large numbers of entities and several hundred packages on disk, example timing improvements are:

area time before changes (ms) time after changes (ms)
Navigation tree 20090 97
Process status viewer 20387 58
Exporting to GIT (exportStatus) 21135 31
Deleting an analytic 20389 7

Kx Stream

TP resilience

The tickerplant has been enhanced to be more resilient to disk outages. This mode is available by enabling the retryInterval instance parameter.

TP resilience

Messaging per-channel publishes

The publish APIs support publishing to all or specific channels. The per-channel publish APIs, .dm.pub*c versions take an extra parameter and only publish to subscribers on that channel. In the example below, the second line will only publish to a subset of the subscribers whereas the first will publish to all.

.dm.pub[`trade; data]
.dm.pubc[`channel; `trade; data]

There is a quirk in this behavior around the null channel `. Null is a valid channel in its own right, however these APIs treat it as a wildcard for all channels. In the example above when using the null channel, both lines would be equivalent.

The more intuitive behavior would be for .dm.pubc to filter on the null channel. To switch to use this behavior, set the following environment variable in your delta.profile.

DELTACONTROL_DMS_USENULL=YES

Reports and deltacomponent

It is now possible to configure the deltacomponent user to be a non-administrator user in Kx Control. If this is the case there may be actions required to grant deltacomponent permission to any solution reports. The administrator status of the deltacomponent user will not change if an existing environment is upgraded to 4.5.0.

Reports and the deltacomponent user

RDB and HDB end of day with secure parser, deltacomponent non-admin

If the deltacomponent user is configued to be a non-administrator user in Kx Control and the secure parser is enabled, there may be issues running end-of-day actions on the HDB from the RDB. This could occur if these actions are defined using string or lambda queries. For an example of how to resolve this see the Kx Control documentation.

RDB and HDB end of day

Kx Monitoring Dashboards queries now Analytics

Kx Monitoring Dashboards have moved all string based queries to offical Analytic APIs. This ensures the APIs are permissioned appropriately and available via secure deploys.

Kx Java Stream API

Extended the configuration of the Java Stream API to support a variety of additional configuration options, available via command-line -D system properties or programmably within the application using the Kx Java Stream API.

This includes - Heartbeating and reconnect configurations of Query Router, Query Manager and Messaging Services - Ability to execute queries against only connected Query Processors - Define Messaging Server queue mode and timeouts

Stream API configurations

Heartbeat log info messages visibility within Java Stream API

Added ability to refresh the client facing public API. This option allows applications to programmatically update the API.

Refresh Public API

Remote subscribers to the the Kx Java Stream API will now recover when connection is severed to the process on a retry timer configured through the system property messaging.server.reconnectratems

Stream API configurations

Kx C# Stream API

Kx C# Stream API now provides an interface to allow an application to request the data be presented as a raw byte stream from the backend KDB+ process. This allows the application to deserialize later/off main thread.

Stream API byte stream support

Kx C# Stream API adds support for certificate authentication and basic instructions on how to import and trust certificates.

Enable TLS certificate verification

Kx Connect

Python script update

The provided example Python script within Kx Connect updated to support both Python 2 and Python 3.

Configurable decimal rounding available

New environmental variable JSON_DECIMAL_FORMAT available to allow the user to configure the JSON format of decimals.

Kx Connect Configuration