Upgrading from Keycloak 16 to Keycloak 18
Keycloak 18 contains changes that are not backwards compatible with Keycloak 16.
The majority of the changes can be handled internally in kdb Insights Enterprise, however there are some changes that must be handled by the user.
In particular, the PostgreSQL database underwent a major version change from version 11 to 14. Existing data must be migrated manually.
Running deployment
The steps documented here assume you have an existing kdb Insights Enterprise deployment of version 1.1.x running.
Variables
Certain variables are referenced throughout this document:
name | description |
---|---|
RELEASE_NAME |
Name of the release |
KEYCLOAK_STATEFULSET |
Name of the Keycloak statefulset |
POSTGRESQL_STATEFULSET |
Name of the PostgreSQL statefulset |
KEYCLOAK_POD |
Name of the Keycloak pod |
POSTGRESQL_PRIMARY_POD |
Name of the PostgreSQL primary pod |
POSTGRESQL_SECRET |
Name of the secret containing PostgreSQL passwords |
POSTGRESQL_PASSWORD |
Password for Postgres |
CHART_REPO |
Name of the Helm repository where kdb Insights Enterprise charts are stored |
CHART |
insights or keycloak-server if a shared keycloak instance is being used |
VERSION |
Version you want to upgrade to |
The variables that must be set explicitly are:
export RELEASE_NAME=<release name> # consult 'helm list' if you are unsure of the release name
export CHART_REPO=<chart repo name>
export CHART=<chart name>
export VERSION=<version>
Kubernetes context
Make sure you are in the correct Kubernetes context when running the commands listed here. This will either be the Insights namespace if you deploy Keycloak as part of Insights, or the Keycloak namespace if you deploy Keycloak separately to kdb Insights Enterprise.
export KEYCLOAK_POD=$(kubectl get pods -l app.kubernetes.io/instance=$RELEASE_NAME,app.kubernetes.io/name=keycloak -o jsonpath="{.items[0].metadata.name}")
export KEYCLOAK_STATEFULSET=$(kubectl get statefulset -l app.kubernetes.io/instance=$RELEASE_NAME,app.kubernetes.io/name=keycloak -o jsonpath="{.items[0].metadata.name}")
export POSTGRESQL_STATEFULSET=$(kubectl get statefulset -l app.kubernetes.io/instance=$RELEASE_NAME,app.kubernetes.io/name=postgresql -o jsonpath="{.items[0].metadata.name}")
export POSTGRESQL_PRIMARY_POD=$(kubectl get pods -l app.kubernetes.io/instance=$RELEASE_NAME,app.kubernetes.io/name=postgresql,role=primary -o jsonpath="{.items[0].metadata.name}")
export POSTGRESQL_SECRET=$(kubectl get pod -l app.kubernetes.io/name=postgresql -o jsonpath='{.items[*].spec.containers[0].env[?(@.name=="POSTGRES_PASSWORD")].valueFrom.secretKeyRef.name}')
export POSTGRESQL_PASSWORD=$(kubectl get secret $POSTGRESQL_SECRET -o jsonpath="{.data.postgresql-password}" | base64 --decode)
Mandatory changes
Postgresql secret format
The secret containing PostgreSQL passwords, usually called kxi-postgresql
, has dropped the postgresql
prefix from the data keys.
You can confirm the name of the secret in your installation by looking at the variable $POSTGRESQL_SECRET
set in the variables section
Keycloak 16:
postgresql-postgres-password
postgresql-password
Keycloak 18:
postgres-password
password
To manually modify the secret:
-
Set the
kubectl.kubernetes.io/last-applied-configuration
annotation on the secret.kubectl get secret $POSTGRESQL_SECRET -o json | kubectl replace --save-config -f -
Note
This step is optional, but it means the old keys will not be removed from the secret
-
Map the old names to the new names and apply the change.
kubectl get secret $POSTGRESQL_SECRET -o json | jq '.data |= with_entries( if .key == "postgresql-postgres-password" then .key = "postgres-password" elif .key == "postgresql-password" then .key = "password" else . end)' | kubectl apply -f -
Values changes
The existingSecret
key in the values file needs to be nested under an auth
key.
Keycloak 16:
keycloak:
postgresql:
existingSecret: kxi-postgresql # replace this with the value of POSTGRESQL_SECRET
Keycloak 18:
keycloak:
postgresql:
auth:
existingSecret: kxi-postgresql # replace this with the value of POSTGRESQL_SECRET
Please modify your values file accordingly.
Database migration
The PostgreSQL databse underwent a major version change between Keycloak 16 and Keycloak 18, from version 11 to 14.
It is necessary to perform a database migration between the two versions, otherwise the PostgreSQL pods will fail to start with an error similar to:
2022-07-14 14:22:34.583 GMT [1] FATAL: database files are incompatible with server
2022-07-14 14:22:34.583 GMT [1] DETAIL: The data directory was initialized by PostgreSQL version 11, which is not compatible with this version 14.3.
The data migration steps are below, these steps follow the pg_dumpall
upgrade steps that are describe in the PostgreSQL documentation here with some modifications based on some suggestions in a Bitnami Github issue here.
Migration steps
Migration failures
If any of the steps below do not complete successfully, please report it to KX and DO NOT proceed with the remaining steps.
Follow the guidance in the rollback section to rollback your changes.
As always, it is advised to test these steps in a development environment before running them on a production database.
-
Scale down Keycloak to no replicas to avoid database changes during the migration.
kubectl scale statefulsets $KEYCLOAK_STATEFULSET --replicas=0
-
Backup the existing database.
kubectl exec $POSTGRESQL_PRIMARY_POD -- bash -c "PGPASSWORD=$POSTGRESQL_PASSWORD pg_dumpall -U postgres" > database_all.sql
-
Delete the PostgreSQL statefulset.
kubectl delete sts $POSTGRESQL_STATEFULSET
-
Delete the gui gateway statefulset.
Shared Keycloak
Skip this step if you are using a shared Keycloak instance
GUI_GATEWAY_STATEFULSET=$(kubectl get statefulset -l app.kubernetes.io/instance=$RELEASE_NAME,app.kubernetes.io/name=gui-gateway -o jsonpath="{.items[0].metadata.name}") kubectl delete sts $GUI_GATEWAY_STATEFULSET
-
Retrieve the values for the release.
helm get values $RELEASE_NAME > values.yaml
-
Upgrade to the new version of PostgreSQL, running in diagnostic mode as root to allow database migration.
Password format changes
Ensure you have migrated to the new password and values format described here before running this command
helm upgrade -f values.yaml --set \ keycloak.postgresql.auth.existingSecret=$POSTGRESQL_SECRET,\ keycloak.postgresql.diagnosticMode.enabled=true,\ keycloak.postgresql.primary.podSecurityContext.runAsNonRoot=false,\ keycloak.postgresql.primary.containerSecurityContext.runAsUser=0,\ keycloak.keycloakConfigCli.enabled=false \ $RELEASE_NAME $CHART_REPO/$CHART --version $VERSION
-
Copy the database dump into the new PostgreSQL pod
tar cf - database_all.sql | kubectl exec -i $POSTGRESQL_PRIMARY_POD -- tar xf - -C /opt/
-
Exec into the new PostgreSQL pod
kubectl exec -it $POSTGRESQL_PRIMARY_POD -- /bin/bash
-
Backup current db instance and init a fresh db instance. These steps have come from the GitHub issue here
. /opt/bitnami/scripts/libos.sh ensure_group_exists postgres -i 1001 ensure_user_exists postgres -i 1001 -g postgres mv /bitnami/postgresql/data /bitnami/postgresql/olddata mkdir -p /bitnami/postgresql/data mkdir -p /bitnami/postgresql/oldbin chown -R postgres:postgres /bitnami/postgresql/data /bitnami/postgresql/olddata cd /tmp/ gosu postgres initdb -E UTF8 -D /bitnami/postgresql/data -U postgres
-
Switch into the
postgres
usersu postgres
-
Start postgres (hit enter after the stdout messages to return to a prompt).
postgres -D /bitnami/postgresql/data &
-
Import the database dump
psql -d postgres -f /opt/database_all.sql
-
Shutdown postgres
pg_ctl stop
-
Exit by running Ctrl+P,Ctrl+Q
-
Run
helm upgrade
again to disable diagnostic mode and running as rootPassword format changes
Ensure you have migrated to the new password and values format described here before running this command
helm upgrade -f values.yaml --set \ keycloak.postgresql.auth.existingSecret=$POSTGRESQL_SECRET \ $RELEASE_NAME $CHART_REPO/$CHART --version $VERSION
Follow on
If you are using a shared Keycloak instance you must upgrade kdb Insights Enterprise deployments that are using the instance to import any realm changes.
If you want to upgrade the operator and CRDs, you can proceed to run kxi install upgrade
Rollback
Pre helm upgrade
If you have not performed any helm upgrade
steps, rollback to the current revision to restore any deleted resources.
You can get the current revision by looking at the output of:
helm ls
Execute the rollback, replacing <current revision>
with the appropriate value.
helm rollback $RELEASE_NAME <current revision>
Post helm upgrade
If you have run any of the helm upgrade
steps, rollback to the previous revision.
helm rollback $RELEASE_NAME
This may not complete successfully if the database was partially migrated.
If this is the case you can follow the restore guide to restore the database from the backup taken as part of the migration.