Skip to content

Perform a dynamic upgrade

This page explains how to deploy a new version of a package to a running KX Sensors environment with a dynamic upgrade (DU): copying it to every node, validating, applying it, verifying, and rolling it back.

The same procedure applies to any new package version, whether it is a release from KX, a new version of your own package, or a hotfix. For the full reference of the commands used here, see Dynamic upgrades using the KX Sensors CLI.

Before you begin

  • The new version must be in its own, versioned directory, with pkg.name and pkg.version set in its .pkg file. Never edit a deployed package in place.
  • Test the deployment in a test or UAT environment first. See Before you begin.
  • Check whether the change needs an offline upgrade instead. KX release notes always say when one is required.

The example

This page uses one example throughout. A three-node system (nodes A, B and C) runs a package example-1.0.0, and you want to upgrade it to example-1.0.1. The starting deployment history is:

$ kxsctl deploy list
Revision  Nodes  Processes/Service Classes  Directory List
--------  -----  -------------------------  --------------------------------------
1         <all>  <all>                      kxs-core-3.0.0;example-1.0.0;env

Note

The kxsctl outputs on this page are illustrative. The values on your system will differ.

Step 1: Copy the package to every node

kxsctl does not distribute packages. The new package directory must exist in the package directory of every node that the deployment targets, and of any node you may target later. If you received the package as an archive, extract it on one host first. Then, from the host that holds the package directory, copy it to every node with rsync (or scp) as the KX Sensors user:

# KXS_ROOT of the target cluster. Set this correctly for the cluster you are
# deploying to: it is not necessarily the same as on the host you copy from.
TARGET_KXS_ROOT=/home/kxs/kxs

for host in host-a host-b host-c; do
  rsync -a example-1.0.1 kxs@${host}:${TARGET_KXS_ROOT}/packages/
done

Every node needs an identical copy

The node where you run kxsctl deploy records the file hashes for the new deployment. If the package directories differ between nodes, the DU may reload the wrong files on some nodes. Always copy from a single source: the package that was tested. Don't rebuild the package in each environment.

Tip

To check that a node's copy matches the source, run rsync -anc --itemize-changes example-1.0.1 kxs@<host>:${TARGET_KXS_ROOT}/packages/. It prints nothing if the copies are identical.

Step 2: Validate the deployment

Confirm that the package directory exists on all nodes. Validation also warns, and asks you to confirm, if EOD is in progress or about to start on a node:

$ kxsctl deploy validate --dirs "example-1.0.1"
Performing pre-DU validations, node=A
Performing pre-DU validations, node=B
Performing pre-DU validations, node=C
Validation successfully completed

If the directory is missing on a node, validation fails and names the node:

Missing package directories on node C: example-1.0.1

Step 3: Apply the new version

Create a new deployment by replacing the current version with the new one in the current directory list, and apply it to the whole system. Because only one package is changing, use --chg (-c) to swap it in the existing directory list, rather than retyping the whole list:

$ kxsctl deploy apply -c "example-1.0.0:example-1.0.1"
Performing pre-DU validations, node=A
Performing pre-DU validations, node=B
Performing pre-DU validations, node=C
This will generate a new deployment derived from revision 1 and apply it to all nodes,
using directory list: kxs-core-3.0.0;example-1.0.1;env

Continue? [yN] y
Process version registry updated, revision=2
Performing post-DU operations, node=A
Performing post-DU operations, node=B
Performing post-DU operations, node=C

Changing more than one package

-c also accepts several old:new pairs separated by semicolons, and can be combined with --add (-a) and --del (-d) to add or remove packages. To redefine the whole directory list instead, use kxsctl deploy set. See Dynamic upgrades using the KX Sensors CLI, in particular Rename a package in an existing deployment and Generate a new deployment.

KX Sensors assigns the new deployment revision 2, and every process moves to it. Each process picks up the new revision from the process version registry and reloads only the files that changed.

A node or process that is down during the deployment starts at its assigned revision when it comes back. This is why the package directory must already be on every node. For details, see How changes propagate.

Rolling out to one node first

To apply the new version to one node first, verify it, and then roll it out to the rest of the system, add -n A to create the deployment for node A only:

kxsctl deploy apply -c "example-1.0.0:example-1.0.1" -n A

After verifying node A as in step 4, apply the existing revision to the whole system, rather than creating a new deployment. This guarantees that every node runs the directory list you verified:

kxsctl deploy apply 2

You can also narrow the target to a service class (-s GW) or a process (-p kxsGW_A). See Perform a partial release.

Step 4: Verify the change

Check that the processes are at revision 2:

$ kxsctl status -v
Node  PID    Status  Name        State   Revision  ...
----  -----  ------  ----------  ------  --------  ...
A     21044  +       kxsGW_A     ACTIVE  2         ...
A     21051  +       kxsRDB_A1   ACTIVE  2         ...
B     18822  +       kxsGW_B     ACTIVE  2         ...
...

The deployment history now shows the new deployment and where it was applied:

$ kxsctl deploy list
Revision  Nodes  Processes/Service Classes  Directory List
--------  -----  -------------------------  --------------------------------------
1         <all>  <all>                      kxs-core-3.0.0;example-1.0.0;env
2         <all>  <all>                      kxs-core-3.0.0;example-1.0.1;env

Then:

  • Check the application logs of the targeted processes. Each process logs the files it detected as changed and ends with Revision change complete. A failure is logged as Upgrade failed in <stage> stage.
  • Confirm that the link on each node now points at the new version: ls -l $KXS_ROOT/packages/example shows example -> example-1.0.1.
  • Test the change itself.

Roll back

To roll back, apply the revision that was in effect before the change. In this example, that is revision 1. The example below rolls back the whole system. To roll back only some nodes, service classes or processes, add the same -n, -s or -p options:

$ kxsctl deploy apply 1
Performing pre-DU validations, node=A
Performing pre-DU validations, node=B
Performing pre-DU validations, node=C
Applying revision 1 to all nodes
Continue? [yN] y
Process version registry updated, revision=1
Performing post-DU operations, node=A
Performing post-DU operations, node=B
Performing post-DU operations, node=C

The rollback appears in the history as another application of revision 1:

$ kxsctl deploy list
Revision  Nodes  Processes/Service Classes  Directory List
--------  -----  -------------------------  --------------------------------------
1         <all>  <all>                      kxs-core-3.0.0;example-1.0.0;env
2         <all>  <all>                      kxs-core-3.0.0;example-1.0.1;env
1         <all>  <all>                      kxs-core-3.0.0;example-1.0.0;env

When you roll back:

  • The changed files are reloaded from example-1.0.0, the configuration cache is cleared, and the example link is repointed to example-1.0.0.
  • Reloading q files redefines what those files define. It does not delete functions or variables that only the new version defined. If the new version added definitions that must not stay active, restart the affected processes after the rollback.
  • Keep the example-1.0.1 directory in place. Revision 2 still references it. If the new version needs correcting, build the correction as a new version. Don't modify example-1.0.1.

You cannot delete a deployment revision. All deployments stay in the history; to undo one, roll back as above.

Caveats

Not every change can be applied safely through a dynamic upgrade. If you are unsure, consult KX and test in UAT first. In particular:

  • feed.yaml and ems.yaml. Dynamic upgrade of these configuration objects is not fully supported. An upgrade of the package that contains them (typically env) must not change them.
  • EOD and DBC. Don't deploy during EOD; pre-DU validation warns if EOD is in progress or about to start. Never deploy a schema change during a DBC (Database Copy) operation.
  • Schema changes. DBW converts the on-disk database during the DU. The rules in About schema changes apply, including the restrictions on which changes can be made dynamically. Some changes need a custom upgrade or downgrade script, which DBW loads during an on-disk conversion at startup; see Use custom upgrade and downgrade scripts. If a package includes such a script, test both the upgrade and the rollback in UAT against a representative copy of the target database.
  • Changes that need a restart. Some changes, such as parameters read only at process startup or definitions that a rollback must remove, only take full effect after the affected processes are restarted. Plan these restarts as part of the deployment and its rollback, using Start and stop service class instances.

Next steps