Skip to content

Perform an offline upgrade

This page explains when offline upgrades are required in KX Sensors and the steps for performing a safe system shutdown and package release.

Unlike dynamic upgrades, offline upgrades require either a complete or partial system shutdown before releasing the new package or schema changes. Offline upgrades are required whenever the change cannot be automated. Examples:

  • You wish to perform non-trivial database table transformations (for example, adding/modifying a column to/in a table where the values for the column are a complex calculation).
  • There are adjustments to how processes are initialized, such as the addition of new functionality within the initialization chain.
  • There are non-trivial configuration changes that can not necessarily be dynamically reloaded (including configurations for which change handlers are not implemented, an example being the EMS configuration used by SP).
  • You wish to upgrade the version of RT (typically will come via an upgrade to the base kxs-core package and noted by KX).
  • You wish to perform an upgrade involving shared objects used by KX Sensors, including the kdb+/q binary (typically will come via an upgrade to the base kxs-core package and noted by KX).
  • There are changes to the interfaces used when communicating between nodes. Example: Adjustments in the GW or KX Sensors comm layer which require adjustment to account for a change in data structure sent from an upgraded node to a non-upgraded node (or vice versa).

These are the known scenarios that require offline upgrades. There may be others depending on the specific changes involved. Upgrades should always be tested prior to applying them in production. Releases which require an offline upgrade will always be called out in release notes by KX.

Make sure that offline conversions are activated in systemParams.yaml before attempting any offline upgrade:

Parameter Description Default
dbConvOffline Whether or not offline upgrades are enabled. 0 = Disabled, 1 = Enabled, 2 = Warn and terminate. 1

Use custom upgrade and downgrade scripts

When DBW is initializing and determines an offline update must take place, it examines the version number of each package and compares it with the version of the last import of that package. If it's determined that a package's version number has increased, then DBW attempts to load a script named <packageName>.upgrade for each package whose version number has increased.

Conversely if it's determined that a package's version has decreased, then DBW attempts to load a script named <packageName>.downgrade for each package whose version number has decreased. Nothing is loaded for packages whose versions have not changed.

These <packageName>.upgrade or <packageName>.downgrade scripts are where you define custom database transformations. This is used for adding/updating a column with some form of complex calculation.

For example, suppose you wish to make a change to your package containing a schema for the table Sample involving modification to the existing integer column testID, adding 100 to each of the IDs. In this case, the conversion can be applied using a custom offline upgrade script as follows:

.conv.applyColFn[`Sample;`testID;-6h;{100i+x`testID};enl`testID]

Note

Your script for upgrading packages should only contain code specific to the conversion to take place. Code from previous conversions should be commented out/removed, or suitable conditional logic added to ensure they will not run again. Make sure that you test your conversion thoroughly before moving to production.

.conv.applyColFn

Adds a transaction to the conversion transaction table specifying some function to apply to a column. This can be used to modify either an existing column or a new column being added as part of the upgrade itself.

Parameter Description
tn Table name.
col Column to change.
typ Data type of column.
fn User-defined function to apply to the table. Should return a list of values conforming to the length of the table to form the new column.
fncols Subset of the columns of the table specifying a section of the table to be fed into the user-defined function.

.conv.delAttr

Adds a transaction to the conversion transaction table specifying a column of a table to remove an attribute from.

Parameter Description
tn Table name.
col Column to change.

Offline upgrade with deployment

This offline upgrade option is like dynamic upgrade with one important difference: you must shut down the system before executing dynamic upgrade commands through the KX Sensors CLI. The standard procedure for an offline upgrade is:

  1. Prepare packages to be deployed via the KX Sensors CLI and ensure they are located within the package directory.
  2. Stop the system using kxsctl stop system.
  3. Execute the required dynamic upgrade commands (consult with previous sections for applications of kxsctl deploy).
  4. If the upgrade involves a schema change, it is recommended to block nodes using the kxsctl block system command. This ensures that DBW will complete on-disk conversion of the database before the remaining processes are started.
  5. Start the system using kxsctl start system.

After starting the system, all processes start using the updated revision number and the updated directory list.

Changes deployed in this manner can similarly be applied to a subset of nodes/service classes/processes, in which case only the nodes of interest need to be stopped/restarted. Depending on the nature of the changes, it may not be necessary to stop the entire node.

Offline upgrade without deployment

If you make changes directly to code/configuration/schemas within an existing package in a deployment, these changes are propagated when relevant processes are restarted. This is not the recommended release procedure as a suitable history of changes may not be readily available. This includes conversion of the on-disk database when DBW is restarted. Proceed with caution and ensure that changes are suitably tested before any upgrade of this style.

RT upgrades

If you are making changes to the version of RT used by KXS, extra precautions should be taken to ensure the version of RT is upgraded correctly. The standard procedure followed is:

  1. Stop RTO using kxsctl stop RTO (use node specific RTOs if upgrade is rolling).
  2. Backup any relevant RT logs (as logs will be removed upon upgrade of RT version).
  3. Perform relevant kxsctl deploy operation to prepare the environment for an upgrade.
  4. Start RTO using kxsctl start RTO (use node specific RTOs if upgrade is rolling).

As of KX Sensors 3.3, the RT implementation ships as a q package rather than as a container image, so this procedure no longer involves loading a new image. The RT logs to back up in step 2 are those under emsLogDir.

Move RT from Docker to the host

Upgrading from 3.2 to 3.3 moves RT out of Docker containers and onto the host. This isn't automated: it's a manual procedure that forms part of the upgrade, and it involves a hard reset of RT, so plan for data loss and schedule it accordingly.

Warning

A hard reset of RT discards RT log and state data. Back up the contents of emsLogDir on every node before you begin, and make sure you understand what's unrecoverable in your deployment.

  1. Stop all nodes in the cluster using kxsctl stop system. Unlike an RT version upgrade, this can't be done as a rolling upgrade — every node must be down.
  2. Perform a hard reset of RT.
  3. Delete the RT Docker containers manually.
  4. Complete the upgrade and restart the system.

You can remove the Docker Swarm overlay network and uninstall Docker at any point during or after the upgrade — neither is required by 3.3, and neither blocks the steps above.

Next steps