Rollback process

This guide details the procedure for rolling back the KX Platform after upgrade. Please note that when an upgrade fails at deploy time the rollback is handled automatically by the install script. This guide focuses on the steps required to rollback an upgraded system to a previously installed version.

How the rollback works

The KX Platform consist of 3 different types of content;

Data

The data generated by the system.

Note: the roll back procedure does not cover rolling back data generated by the system since the upgrade as this is solution specific and so outside of the scope of this document

Environment config

Applications and Binaries: These are configured via Control entities but also by the delta.profile, delta.instance.profile, delta_user.profile files which define the path to the current version of Control, C++ Binaries and Java Programs along with their port/host/taskset settings.

Licenses / Scripts: These are located in the delta-bin/config directory and in the deltabin/bin dir. They include the current license files, SSL certificates and the current start/stop scripts.

KX Control data

KX Control stores its internal state in a set of kdb+ tables in a directory called tdir (referenced via the DELTACONTROL_TDIR in delta.profile). This contains the current version of all of the entities inside Control and this is where Control loads its state from when it is started.

In order to roll back successfully each of the content types in 2) and 3) need to be reverted to the state they were in pre-upgrade. An overview of the rollback procedure for each of these content types is given below:

Environment config

Rolling back the Environment Config involves identifying the backup files and directories created by the install script during the upgrade and using them to revert the current active versions.

KX Control data

Control tdir data is rolled back using the tdirarchive directory which contains a TGZ archive of Control’s state created each time Control is cleanly shutdown.

Control qsrc is rolled back using the qsrcarchive directory which contains TGZ archives of the qsrc files created each time Control is cleanly shutdown.

Rollback pre-requisites

To roll back the environment you will need the following;

Upgrade details

To roll back you must identify the last upgrade date and time. If you have the installation logfile (e.g. installDeltaXML.20180301_111047.log) from the upgrade you will be able to see where the script has backed up the Environment Config detailed in the previous section of this document.

The backup location for packages and config changed in version 4.157 of the installDeltaXML.sh script. If you are using an older version of the script you can find the locations from the log file using the following commands:

grep "create backup file" installDeltaXML.20180215_105352.log
create backup file [kxinstall/delta-bin/.bak/delta.profile.20180215_105352]
create backup file [kxinstall/delta-bin/.bak/delta.instance.profile.20180215_105352]
create backup file [kxinstall/delta-bin/.bak/delta_user.profile.20180215_105352]

grep “create backup directory” installDeltaXML.20180215_105352.log
create backup directory [/home/ddempsey/kxinstall/delta-bin/.bak/bin.20180215_105352]

For script version 4.157 and later the locations can be found using the following command:

grep "create backup file" installDeltaXML.20180321_160129.logcreate backup file [
create backup file [kxinstall/backup/20180321_160129/qsrc/qsrc_2250_v1_2018.03.21T160124.tgz]
create backup file [kxinstall/backup/20180321_160129/tdir/tdir_2250_v1_2018.03.21T160124.tgz]
create backup file [kxinstall/backup/20180321_160129/profiles/delta.profile]
create backup file [kxinstall/backup/20180321_160129/profiles/delta.instance.profile]
create backup file [kxinstall/backup/20180321_160129/profiles/delta_user.profile]

Note: for version 4.157 and later the location of the qsrc and tdir files are also given in the output above.

The install timestamp can also be found by looking in delta-bin/.bak (version 4.156 and earlier) and delta-bin/backup (version 4.157 and later).

qsrc archive

The qsrcarchive directory (delta-data/DeltaControlData/qsrcarchive) contains TGZ archives of the qsrc (process templates). The archives are created automatically each time Control is shutdown. The process templates differ between versions of Control and Stream so to rollback you need to identify the archive version which was created just prior to the upgrade.

You can find out the location of the previous qsrc archive from the install log (version 1.456 and earlier)

grep "previous qsrc archive" installDeltaXML.20180215_105352.log
previous qsrc archive [qsrc_2250_v2_2018.02.15T105348.tgz]

tdir archive

The tdirarchive directory (delta-data/DeltaControlData/tdirarchive) contains archives of the tdir (control state tables). The archives are created automatically each time Control is shutdown. To rollback an upgrade you will also need to rollback the tdir so that all the control entities from stream and solution packages are reverted.

You can find out the location of the previous tdir archive from the install log (version 1.456 and earlier)

grep "previous tdir archive" installDeltaXML.20180215_105352.log
previous tdir archive [tdir_2250_v2_2018.02.15T105347.tgz]

Tomcat

Web applications (Dashboards/Connect/Analyst etc) are deployed into the Tomcat webapps directory. During an upgrade if Tomcat itself is not being deployed then the previous versions of these webapps are tarred up inside the webapps directory. (script version 4.146 and earlier)

ls -1 kxinstall/delta-bin/software/Tomcat_7_0_73/latest/webapps/
analyst
analyst.20180215_130927.tar.gz
connect
connect.20180215_130927.tar.gz
control
control.20180215_130927.tar.gz
dashboards
dashboards.20180215_130927.tar.gz
delta
delta.20180215_130927.tar.gz
ROOT
ROOT.20180215_130927.tar.gz

When rolling back an install you will need to identify the tarballs that were created during the upgrade. You can do this by listing the contents of the webapps directory as above or if you have the install log from the upgrade you can get the TGZ names as follows:

grep "tarring up current web application" installDeltaXML.20180215_132727.log
tarring up current web application [ROOT] to [ROOT.20180215_132727.tar.gz] in
[kxinstall/delta-bin/software/Tomcat_7_0_73/latest/webapps]
tarring up current web application [delta] to [delta.20180215_132727.tar.gz] in
[kxinstall/delta-bin/software/Tomcat_7_0_73/latest/webapps]
tarring up current web application [connect] to [connect.20180215_132727.tar.gz] in
[kxinstall/delta-bin/software/Tomcat_7_0_73/latest/webapps]
tarring up current web application [dashboards] to [dashboards.20180215_132727.tar.gz] in
[kxinstall/delta-bin/software/Tomcat_7_0_73/latest/webapps]
tarring up current web application [control] to [control.20180215_132727.tar.gz] in
[kxinstall/delta-bin/software/Tomcat_7_0_73/latest/webapps]

If the upgrade was performed using version 4.147 or later the entire webapps directory is backed up into the backup directory along with the system configurations:

ls -1 backup/20180321_160129/
bin
config
profiles
qsrc
tdir
webapps

If tomcat itself was upgraded during an upgrade then the new webapps will be deployed into the directory of the new tomcat. This will remove any manual rollback steps specific to web applications.

Rollback procedure - single server deploy

The following section gives the steps required to rollback a platform upgrade using an example where KX Platform version 4.0.1 has been upgraded to version 4.0.2.

Before proceeding with the rollback identify the backup files from the following section. For this example the backup files are detailed in the tables below:

Environment config	Backup
delta.profile	kxinstall/backup/<date><time>/profiles/delta.profile
delta_user.profile	kxinstall/backup/<date><time>/profiles/delta_user.profile
delta.instance.profile	kxinstall/backup/<date><time>/profiles/delta.instance.profile
bin	kxinstall/backup/<date><time>/bin

Control Data	Backup
qsrc	kxinstall/backup/<date><time>/qsrc/qsrc_2001_v212_2018.03.13T100617.tgz
tdir	kxinstall/backup/<date><time>/tdir/tdir_2001_v212_2018.03.13T100616.tgz

Web Applications	Backup
webapps	kxinstall/backup/<date><time>/webapps

Rollback steps

1) Shutdown all running processes

$ cd kxinstall/delta-bin/bin/
$ ./stopAllProcesses.sh

2) Shutdown Control

$ cd kxinstall/delta-bin/bin/
$ ./stopDeltaControl.sh

3) Check for any remaining running processes using ps

$ ps ux | grep -e "q " -e "java"

4) Revert environment config from backup

$ cd kxinstall/delta-bin/
$ cp ../backup/20180313_101225/profiles/delta.profile delta.profile
$ cp ../backup/20180313_101225/profiles/delta_user.profile delta_user.profile
$ cp ../backup/20180313_101225/profiles/delta.instance.profie delta.instance.profile
$ rm -rf bin/*
$ cp ../backup/20180313_101225/bin/* bin/

5) Revert Control data from backup

$ cd kxinstall/delta-data/DeltaControlData
$ mv qsrc qsrc.$(date +%Y%m%d)
$ mv tdir tdir.$(date +%Y%m%d)
$ cd kxinstall/backup/20180313_101225
$ tar -xvzf qsrc/qsrc_2001_v212_2018.03.13T100617.tgz –C /
$ tar -xvzf tdir/tdir_2001_v212_2018.03.13T100616.tgz -C /

6) Revert Web applications from backup

Note: This step only applies if Tomcat was not deployed during the upgrade.

$ cd kxinstall/delta-bin/
$ . delta.profile
$ rm -rf $DELTA_WEBAPPS
$ cd kxinstall/backup/20180313_101225
$ cp -r webapps $DELTA_WEBAPPS

7) Bring up system back up

Rollback procedure - control cluster deploy

The process for rolling back a clustered deploy is the same as that used for the Single Server deploy in the previous section. The only difference is that the rollback steps have to be executed on each of the hosts in the Control cluster. Backup files need to be identified on all hosts in the Control cluster before proceeding with the Rollback.

Note it is important that all Platform processes are down on all Cluster nodes before performing the rollback steps. See the Linux Administration section of this guide for more details.