Upgrading kdb Insights Enterprise
This section details the steps to upgrade kdb Insights Enterprise from a current install.
If you are upgrading kdb Insights Enterprise on Azure Marketplace, reach out to KX Support.
Prerequisites
Pre-flight checks
The upgrade process requires downtime to ensure data integrity can be guaranteed. This means that before any upgrade is scheduled, a number of pre-flight checks are necessary.
-
Ensure the latest version of kxi-cli is installed
-
Pre-Upgrade Versions
The currently running versions of kdb Insights Enterprise and Operator should be noted for a rollback scenario
-
Operator
helm ls -n kxi-operator
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION insights kxi-operator 1 2022-04-01 15:39:44.955236379 +0000 UTC deployed kxi-operator-1.0.0 1.0.0
Overriding the Operator version
The CLI installs the latest kdb Insights Operator minor version that matches the specified
$INSIGHTS_VERSION
. You can override this by passing the--operator-version
command line option. -
Application
helm ls -n $INSIGHTS_NAMESPACE
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION insights kxi 1 2022-04-01 16:57:00.144126754 +0000 UTC deployed insights-1.0.0 1.0.0
-
kubectl
You must have the kubectl tool installed on your machine and configured to access the same cluster as the kdb Insights Enterprise deployment you are upgrading.
Variables
Have the following information to hand:
variable | example | further details |
---|---|---|
INSIGHTS_NAMESPACE | kxi | Namespace of existing deployment |
INSIGHTS_VERSION | 1.13.0 | Version to upgrade to |
INSTALL_CONFIG_FILE | values.yaml | Install configuration |
PRE_UPGRADE_INSIGHTS_VERSION | 1.12.0 | Current application version |
PRE_UPGRADE_OPERATOR_VERSION | 1.12.0 | Current Operator version |
Retrieve install configuration
Retrieve the INSTALL_CONFIG_FILE
from the existing installation with the following command.
kxi install get-values --namespace $INSIGHTS_NAMESPACE > $INSTALL_CONFIG_FILE
Configure a default profile
If you are upgrading to kdb Insights Enterprise version 1.13 or higher, you must configure a default profile for the environment you are upgrading, or you must specify a profile override.
Other version-specific considerations
Refer to the release notes for any specific upgrade considerations for your version of kdb Insights Enterprise. This may include customizing the install configuration file retrieved previously.
Stop ingestion feeds
Stop all feeds into the application to avoid data loss with inflight data.
Upgrade with CLI
Upgrading using the CLI does not remove the persisted data backing kdb Insights Enterprise configuration or databases. These remain as persisted volumes within the Kubernetes cluster.
The steps performed by the CLI are:
- Backup packages and any running assemblies
- Teardown packages
- Upgrade application, Operator and CRDs
- Convert assemblies that were running prior to the upgrade to packages, if you have chosen to convert them
- Restore the packages
Note
The use of kxi assembly
to deploy assembly yaml files has been deprecated. As part of the upgrade, you are prompted to choose if you want to convert any running assemblies to packages and redeploy using kxi pm
after the upgrade.
If you choose not to convert these assemblies, the CLI does not redeploy them after the upgrade, they must be manually deployed.
If there are no packages or assemblies running, the CLI upgrades the base system and skips the package and assembly steps.
To upgrade with a previously configured default profile, run the following command which steps through the upgrade, prompting you as required.
kxi install upgrade --filepath $INSTALL_CONFIG_FILE --version $INSIGHTS_VERSION
Backing up packages
Persisted package artifact for kxi-db to <.kxi BACKUP FILE>
Persisted package artifact for insights-demo to <.kxi BACKUP FILE>
...
Upgrading insights and operator
...
Reapplying packages
...
Upgrade to version x.x complete
...
Note
The previous example shows two sample packages; you may have different packages, or none at all.
To upgrade with a profile override, run:
kxi --profile <profile name> install upgrade --version 1.13.0 --operator-version 1.13.0 --filepath values.yaml
Configuring a profile
If you want to configure a new profile, run kxi configure --profile <name of new profile>
command.
Encryption in Transit
If you have not previously set encryption of data in transit, you will be prompted to enable it during upgrade.
If you have previously enabled encryption for data in transit, the default behavior is to keep the existing, pre-upgrade setting.
To change this setting, add the --encryption true/false
command line parameter.
Post-upgrade checks
Check the system is running correctly.
-
Expected version of kdb Insights Enterprise is running
kubectl describe cm insights-config | grep version version: insights-1.1.1
-
Operator installed
helm ls -n kxi-operator
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION insights kxi-operator 1 2022-04-11 15:39:44.955236379 +0000 UTC deployed kxi-operator-1.1.0 1.1.0
-
Application installed
helm ls -n $INSIGHTS_NAMESPACE
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION insights kxi 1 2022-04-11 16:57:00.144126754 +0000 UTC deployed insights-1.1.1 1.1.1
-
Operator in running state
kubectl get pods -n kxi-operator
NAME READY STATUS RESTARTS AGE insights-kxi-operator-6df4bcfddc-gnnjj 1/1 Running 0 85m insights-kxi-operator-6df4bcfddc-lqmrl 1/1 Running 0 85m insights-kxi-operator-6df4bcfddc-z45wq 1/1 Running 0 85m
-
Application in running state
kubectl get pods -n $INSIGHTS_NAMESPACE
NAME READY STATUS RESTARTS AGE insights-aggregator-9959bdc87-4pg7z 2/2 Running 0 9m47s ... ... insights-sg-gateway-596b486d9f-pmhf4 1/1 Running 0 9m47s insights-sg-gateway-596b486d9f-qvzbr 1/1 Running 0 9m47s
-
Assembly CRDs redeployed
kubectl get crds | grep insights.kx.com
assemblies.insights.kx.com 2022-04-11T15:23:53Z assemblyresources.insights.kx.com 2022-04-11T15:23:54Z
-
Ensure packages are up and running successfully. The following example shows two sample packages; you may have different packages, or none at all.
kxi pm list
╭────────────-──┬─────────┬────────┬─────────────────────┬──────────────────────────────────────╮ │ name │ version │ access │ owner.name │ id │ ├─────────────-─┼─────────┼────────┼─────────────────────┼──────────────────────────────────────┤ │ kxi-db │ 1.0.0 │ ARWX │ │ 529ab8df-3f1a-4b5b-8d69-29ecee6ca2f5 │ │ insights-demo │ 0.0.1 │ ARWX │ │ add333ad-bdd2-41a9-bcc7-a24fdbee598f │ ╰──────────────-┴─────────┴────────┴─────────────────────┴──────────────────────────────────────╯
If all post-upgrade checks pass, you can restart any feeds into the application that were stopped before the upgrade.
Rollback
You can use the kxi install rollback
command to roll back to an older version of kdb Insights Enterprise.
-
Run
kxi install history
to review the release history and choose which revision you want to roll back to:$ kxi install history --show-operator REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Tue Feb 28 14:10:03 2023 deployed insights-1.3.0 1.3.0 Install complete 2 Tue Feb 28 14:16:41 2023 failed insights-1.4.0 1.4.0 Upgrade complete 3 Tue Feb 28 14:25:55 2023 failed insights-1.4.1 1.4.1 Upgrade "insights" failed: post-upgrade hooks failed: timed out waiting for the condition 1 Tue Feb 28 14:09:52 2023 superseded kxi-operator-1.3.1 1.3.1 Install complete 2 Tue Feb 28 14:15:37 2023 superseded kxi-operator-1.4.0 1.4.0 Upgrade complete 3 Tue Feb 28 14:24:54 2023 superseded kxi-operator-1.4.2 1.4.2 Upgrade complete
If you want only kdb Insights Enterprise revisions, remove the
--show-operator
from the above command.Revision Value
To initiate a rollback, it's important to note that the rollback command relies on the REVISION value rather than the APP VERSION. In order to revert back to version 1.3.0 of kdb Insights Enterprise, you must identify the correct REVISION number, which in this case is simply 1.
The version of kdb Insights Operator and kdb Insights Enterprise must match
If the currently installed version of kdb Insights Operator does not align with the rollback version of kdb Insights Operator, it's necessary to perform the rollback on both.
-
Teardown all running packages.
- Perform Database Rollback.
-
Run
kxi install rollback
with the chosen revisions:$ kxi install rollback $INSIGHTS_REVISION --operator-revision $OPERATOR_REVISION Rolling Insights back to version 1.12.0 and revision 1. And operator back to version 1.12.0 and revision 1 [y/N]: y Backing up packages No packages to back up ...
No explicit revision
If you do not provide an explicit revision,
kxi install rollback
will choose the most recent one.Rolling back kdb Insights Enterprise only
Omit
--operator-revision
if you want to rollback only kdb Insights Enterprise. -
Once the rollback has completed, deploy each torn down assembly and package.
Where your upgrade attempt was from any 1.12 version or below, you can redeploy your assembly by executing a command such as:
kxi assembly deploy --filepath kxi-assembly.yaml
Packages can be redeployed by executing a command such as:
kxi pm deploy kxi-db
Database Rollback
If you are performing a rollback from 1.13.x to an earlier version, assemblies must not be running prior to initiating the rollback. This process ensures no data is lost.
- Teardown your package(s).
- Run the following command to get the current version of kxi-db-rollback:
helm search repo kx-insights/kxi-db-rollback --versions
-
Run the
kxi-db-rollback
chart with the following command for package one, substituting the desired values:helm install rollback kx-insights/kxi-db-rollback --version="[current kxi-db-rollback version]" --set rollback.currentVersion="[current kxi-db-rollback version]" --set rollback.version="[Insights version to rollback to]" --set rollback.assembly="[your-package-name]" -f[your values.yaml file]
The parameters used are: - `version` - the chart version, which dictates the version of the rollback image being used - `rollback.version` - the version you want to rollback to - `rollback.currentVersion` - your current version - this is likely the same as `version` - `rollback.assembly` - the name of the database package
For example, if you want to rollback from version
1.13.0
to1.12.7
export CURRENT_VERSION=1.13.0 export ROLLBACK_TO_VERSION=1.12.7 helm install rollback kx-insights/kxi-db-rollback \ --version="$CURRENT_VERSION" \ --set rollback.currentVersion="$CURRENT_VERSION" \ --set rollback.version="$ROLLBACK_TO_VERSION" \ --set rollback.assembly="[your-package-name]" \ -f[your values.yaml file]
Omitting version parameter
If
version
is omitted, then the latest version of the chart from your repo will be used. It is safer to explicitly set the version so that it contains all the necessary rollback commands. -
Do one of the following:
- If you have only one package, go to Step 5.
- If you have more than one package, uninstall the
kxi-db-rollback
chart by running thehelm uninstall rollback
command. Then, repeat Step 3 for the second package. Next, run thehelm uninstall rollback
command again. Repeat Step 3 for the third package. Continue, until all packages have been processed, runninghelm uninstall rollback
between each invocation, and then repeating Step 3 for each package.
Note
Step 3 is required for each package of your installation, including any packages which weren't deployed.
-
Redeploy the package(s) which you tore down.
Failed upgrades
The following is an example output from a failed upgrade attempt from version 1.12.0, where only assemblies were running before the upgrade.
...
Persisted assembly definitions for ['dfx-assembly', 'iot-assembly'] to /tmp/tmp7ny84esa-kxi-assembly-state.yaml
...
Error: Command '['helm', 'upgrade', '--install', '-f', './values.yaml', 'insights', 'kx-insights/insights', '--version', '1.13.0', '--namespace', 'insights']' returned non-zero exit status 1.
State files are preserved in the event of an upgrade failure.
After recovering the state of kdb Insights Enterprise, you can restore state files as follows.
Where your upgrade attempt was from any 1.12 version or below, you can restore assembly state files to their running state by executing the following command:
kxi assembly create --filepath /tmp/tmp7ny84esa-kxi-assembly-state.yaml
Packages can be restored using package manager by executing a command such as:
kxi pm deploy kxi-db
Updating your application configuration
You can apply advanced configurations to your running instance of kdb Insights Enterprise by upgrading with an updated install config file.
-
Retrieve your install configuration from the existing application and update the resulting
INSTALL_CONFIG_FILE
. -
Upgrade your installation with the updated values file, passing the same version number as the running installation.