Upgrading kdb Insights Enterprise on Azure

This page explains how to upgrade kdb Insights Enterprise, and your Kubernetes version on Azure.

KX Managed

If you deployed kdb Insights Enterprise using the Azure Marketplace the Insights Managed Service (IMS) team will reach out to you to arrange a maintenance window for upgrades. IMS takes care of upgrading the kdb Insights Enterprise, the third party dependencies and the kubernetes version of your AKS cluster.

If you have any questions please raise a ticket.

Rook-Ceph

If you deployed kdb Insights Enterprise through the Azure Marketplace, whether using the Managed or Unmanaged offering, upgrading Rook Ceph from version 1.17 to 1.18 or later may result in known issues. Please refer to the upgrade guidance provided at the bottom of this page before proceeding.

Azure Marketplace Monitoring Stack

Azure Marketplace deployments could use their own monitoring stack, relying on Azure Monitor and Azure Log Analytics. We recommend you use Prometheus and Grafana which can be installed using the kxi install monitoring command. When changing monitoring stack be sure to disable the Azure monitoring stack by executing:

az aks disable-addons -a monitoring -n "$CLUSTER_NAME" -g "$RESOURCE_GROUP"

Upgrading AKS cluster

Rook-Ceph

If you have deployed rook-ceph, please do not use the Azure AKS rolling upgrade feature. There are known issues with this. Please reach out to KX Support team for assistance.

Upgrading Kubernetes on kdb Insights Enterprise on Azure

This section explains how to upgrade your Kubernetes version, while keeping your kdb Insights Enterprise running on Azure.

The Kubernetes version can be upgraded automatically or manually. Learn more about upgrading your AKS cluster

Prerequisite

To perform a Kubernetes upgrade for kdb Insights Enterprise version 1.12.0 or higher, the following manual process needs to be performed (regardless of your upgrade choice - automated or manual).

Check that you have sufficient CPU quota available in your Azure subscription to create a new node per each node pool.

The Istio ReplicaSet needs to be scaled from 1 to 3:

Connect to the kdb Insights Enterprise cluster on Azure.
```
az account set --subscription <SUBSCRIPTION>
```

Get Azure cluster credentials

az aks get-credentials --resource-group <RESOURCE GROUP> --name <KXI KUBERNETES CLUSTER NAME> --overwrite-existing

Set the minReplicas value to 3 for the Istio HorizontalPodAutoscaler (HPA)

kubectl patch hpa istiod -n istio-system --type=merge -p '{"spec":{"minReplicas":3}}'

To prevent the upgrade from becoming stuck, increase the Pod disruption budgets (PDBs) from 0 to 1 where applicable:
1. Set the MODIFIED_PDBS_FILE environment variable to specify the file path where all patched PDBs are stored. If this variable is not set, the default path of /tmp/modified_pdbs.txt is used.
2. Download the patch_pdb.sh script file which updates PDBs to allow pod disruption during the upgrade.
3. Navigate to the folder where the script was downloaded, make it executable, and run it.
```
chmod +x patch_pdb.sh
./patch_pdb.sh
```

Upgrade

If the cluster requires a manual update, do the following:
- Read the documentation: Upgrade an AKS cluster.
- Access the Azure portal.
- Navigate to kdb Insights Enterprise Azure Kubernetes Service > Settings > Cluster Configuration.
- Click Upgrade version to trigger the manual update of Kubernetes.
Monitor the progress of the upgrade:
- Check the node pools.
- Check the workloads.

Check if all nodes run the desired version.

kubectl get nodes -o json | jq -r '.items[] | "\(.status.nodeInfo.kubeletVersion) \(.metadata.name)"'

Post-Upgrade (optional)

Set the minReplicas value back to 1 for the Istio HPA:

kubectl patch hpa istiod -n istio-system --type=merge -p '{"spec":{"minReplicas":1}}'

If the post-upgrade tasks are run in a different shell than the prerequisite tasks, set the MODIFIED_PDBS_FILE environment variable to the same value that was used before. If this variable is not set, the default path of /tmp/modified_pdbs.txt is used.
Download the revert_pdb.sh script file which reverts the PDBs to disallow pod disruption.
Navigate to the folder where the script was downloaded, make it executable, and run it.
```
chmod +x revert_pdb.sh
./revert_pdb.sh
```

Upgrading Rook-Ceph to 1.18+ on Azure

When upgrading Rook Ceph alongside the main upgrade, the following issue may occur and requires manual intervention.

Issue: The CSI driver tolerations may not be properly configured after upgrading Rook Ceph, potentially causing scheduling issues on certain nodes.

Resolution: Apply the following patch to the Rook Ceph CSI driver:

kubectl patch driver.csi.ceph.io rook-ceph.cephfs.csi.ceph.com -n rook-ceph --type='merge' -p '
{
  "spec": {
    "nodePlugin": {
      "tolerations": [
        {
          "effect": "NoSchedule",
          "key": "CriticalAddonsOnly",
          "operator": "Exists"
        }
      ]
    }
  }
}'

This command adds the required toleration to allow the CSI node plugin to run on nodes with the CriticalAddonsOnly taint.