Skip to content

Upgrading from Keycloak 16 to Keycloak 18

Keycloak 18 contains changes that are not backwards compatible with Keycloak 16.

The majority of the changes can be handled internally in Insights, however there are some changes that must be handled by the user.

In particular, the PostgreSQL database underwent a major version change from version 11 to 14. Existing data must be migrated manually.

Running deployment

The steps documented here assume you have an existing KX Insights deployment of version 1.1.x running.

Variables

Certain variables are referenced throughout this document:

name description
RELEASE_NAME Name of the release
KEYCLOAK_STATEFULSET Name of the Keycloak statefulset
POSTGRESQL_STATEFULSET Name of the PostgreSQL statefulset
KEYCLOAK_POD Name of the Keycloak pod
POSTGRESQL_PRIMARY_POD Name of the PostgreSQL primary pod
POSTGRESQL_SECRET Name of the secret containing PostgreSQL passwords
POSTGRESQL_PASSWORD Password for Postgres
CHART_REPO Name of the Helm repository where KX Insights charts are stored
CHART insights or keycloak-server if a shared keycloak instance is being used
VERSION Version you want to upgrade to

The variables that must be set explicitly are:

export RELEASE_NAME=<release name> # consult 'helm list' if you are unsure of the release name
export CHART_REPO=<chart repo name>
export CHART=<chart name>
export VERSION=<version>

Kubernetes context

Make sure you are in the correct Kubernetes context when running the commands listed here. This will either be the Insights namespace if you deploy Keycloak as part of Insights, or the Keycloak namespace if you deploy Keycloak separately to Insights.

export KEYCLOAK_POD=$(kubectl get pods -l app.kubernetes.io/instance=$RELEASE_NAME,app.kubernetes.io/name=keycloak -o jsonpath="{.items[0].metadata.name}")
export KEYCLOAK_STATEFULSET=$(kubectl get statefulset -l app.kubernetes.io/instance=$RELEASE_NAME,app.kubernetes.io/name=keycloak -o jsonpath="{.items[0].metadata.name}")
export POSTGRESQL_STATEFULSET=$(kubectl get statefulset -l app.kubernetes.io/instance=$RELEASE_NAME,app.kubernetes.io/name=postgresql -o jsonpath="{.items[0].metadata.name}")
export POSTGRESQL_PRIMARY_POD=$(kubectl get pods -l app.kubernetes.io/instance=$RELEASE_NAME,app.kubernetes.io/name=postgresql,role=primary -o jsonpath="{.items[0].metadata.name}")
export POSTGRESQL_SECRET=$(kubectl get pod -l app.kubernetes.io/name=postgresql -o jsonpath='{.items[*].spec.containers[0].env[?(@.name=="POSTGRES_PASSWORD")].valueFrom.secretKeyRef.name}')
export POSTGRESQL_PASSWORD=$(kubectl get secret $POSTGRESQL_SECRET -o jsonpath="{.data.postgresql-password}" | base64 --decode)

Mandatory changes

Postgresql secret format

The secret containing PostgreSQL passwords, usually called kxi-postgresql, has dropped the postgresql prefix from the data keys.

You can confirm the name of the secret in your installation by looking at the variable $POSTGRESQL_SECRET set in the variables section

Keycloak 16:

  • postgresql-postgres-password
  • postgresql-password

Keycloak 18:

  • postgres-password
  • password

To manually modify the secret:

  1. Set the kubectl.kubernetes.io/last-applied-configuration annotation on the secret.

    kubectl get secret $POSTGRESQL_SECRET -o json | kubectl replace --save-config -f -
    

    Note

    This step is optional, but it means the old keys will not be removed from the secret

  2. Map the old names to the new names and apply the change.

    kubectl get secret $POSTGRESQL_SECRET -o json | jq '.data |=
        with_entries(
          if .key == "postgresql-postgres-password"
          then
            .key = "postgres-password"
          elif
            .key == "postgresql-password"
          then
            .key = "password"
          else
            .
          end)' | kubectl apply -f -
    

Values changes

The existingSecret key in the values file needs to be nested under an auth key.

Keycloak 16:

keycloak:
  postgresql:
    existingSecret: kxi-postgresql # replace this with the value of POSTGRESQL_SECRET

Keycloak 18:

keycloak:
  postgresql:
    auth:
      existingSecret: kxi-postgresql # replace this with the value of POSTGRESQL_SECRET

Please modify your values file accordingly.

Database migration

The PostgreSQL databse underwent a major version change between Keycloak 16 and Keycloak 18, from version 11 to 14.

It is necessary to perform a database migration between the two versions, otherwise the PostgreSQL pods will fail to start with an error similar to:

2022-07-14 14:22:34.583 GMT [1] FATAL:  database files are incompatible with server
2022-07-14 14:22:34.583 GMT [1] DETAIL:  The data directory was initialized by PostgreSQL version 11, which is not compatible with this version 14.3.

The data migration steps are below, these steps follow the pg_dumpall upgrade steps that are describe in the PostgreSQL documentation here with some modifications based on some suggestions in a Bitnami Github issue here.

Migration steps

Migration failures

If any of the steps below do not complete successfully, please report it to KX and DO NOT proceed with the remaining steps.

Follow the guidance in the rollback section to rollback your changes.

As always, it is advised to test these steps in a development environment before running them on a production database.

  1. Scale down Keycloak to no replicas to avoid database changes during the migration.

    kubectl scale statefulsets $KEYCLOAK_STATEFULSET --replicas=0
    
  2. Backup the existing database.

    kubectl exec $POSTGRESQL_PRIMARY_POD -- bash -c "PGPASSWORD=$POSTGRESQL_PASSWORD pg_dumpall -U postgres" > database_all.sql
    
  3. Delete the PostgreSQL statefulset.

    kubectl delete sts $POSTGRESQL_STATEFULSET
    
  4. Delete the gui gateway statefulset.

    Shared Keycloak

    Skip this step if you are using a shared Keycloak instance

    GUI_GATEWAY_STATEFULSET=$(kubectl get statefulset -l app.kubernetes.io/instance=$RELEASE_NAME,app.kubernetes.io/name=gui-gateway -o jsonpath="{.items[0].metadata.name}")
    kubectl delete sts $GUI_GATEWAY_STATEFULSET
    
  5. Retrieve the values for the release.

    helm get values $RELEASE_NAME > values.yaml
    
  6. Upgrade to the new version of PostgreSQL, running in diagnostic mode as root to allow database migration.

    Password format changes

    Ensure you have migrated to the new password and values format described here before running this command

    helm upgrade -f values.yaml --set \
    keycloak.postgresql.auth.existingSecret=$POSTGRESQL_SECRET,\
    keycloak.postgresql.diagnosticMode.enabled=true,\
    keycloak.postgresql.primary.podSecurityContext.runAsNonRoot=false,\
    keycloak.postgresql.primary.containerSecurityContext.runAsUser=0,\
    keycloak.keycloakConfigCli.enabled=false \
    $RELEASE_NAME $CHART_REPO/$CHART --version $VERSION
    
  7. Copy the database dump into the new PostgreSQL pod

    tar cf - database_all.sql | kubectl exec -i $POSTGRESQL_PRIMARY_POD -- tar xf - -C /opt/
    
  8. Exec into the new PostgreSQL pod

    kubectl exec -it $POSTGRESQL_PRIMARY_POD -- /bin/bash
    
  9. Backup current db instance and init a fresh db instance. These steps have come from the GitHub issue here

    . /opt/bitnami/scripts/libos.sh
    ensure_group_exists postgres -i 1001
    ensure_user_exists postgres -i 1001 -g postgres
    mv /bitnami/postgresql/data /bitnami/postgresql/olddata
    mkdir -p /bitnami/postgresql/data mkdir -p /bitnami/postgresql/oldbin
    chown -R postgres:postgres /bitnami/postgresql/data /bitnami/postgresql/olddata
    cd /tmp/
    gosu postgres initdb -E UTF8 -D /bitnami/postgresql/data -U postgres
    
  10. Switch into the postgres user

    su postgres
    
  11. Start postgres (hit enter after the stdout messages to return to a prompt).

    postgres -D /bitnami/postgresql/data &
    
  12. Import the database dump

    psql -d postgres -f /opt/database_all.sql
    
  13. Shutdown postgres

    pg_ctl stop
    
  14. Exit by running Ctrl+P,Ctrl+Q

  15. Run helm upgrade again to disable diagnostic mode and running as root

    Password format changes

    Ensure you have migrated to the new password and values format described here before running this command

    helm upgrade -f values.yaml --set \
    keycloak.postgresql.auth.existingSecret=$POSTGRESQL_SECRET \
    $RELEASE_NAME $CHART_REPO/$CHART --version $VERSION
    

Follow on

If you are using a shared Keycloak instance you must upgrade Insights deployments that are using the instance to import any realm changes.

If you want to upgrade the operator and CRDs, you can proceed to run kxi install upgrade

Rollback

Pre helm upgrade

If you have not performed any helm upgrade steps, rollback to the current revision to restore any deleted resources.

You can get the current revision by looking at the output of:

helm ls

Execute the rollback, replacing <current revision> with the appropriate value.

helm rollback $RELEASE_NAME <current revision>
Post helm upgrade

If you have run any of the helm upgrade steps, rollback to the previous revision.

helm rollback $RELEASE_NAME

This may not complete successfully if the database was partially migrated.

If this is the case you can follow the restore guide to restore the database from the backup taken as part of the migration.