Managing secrets on Azure

It may be necessary to change or recreate secrets stored in AKS. Typical use cases are as follows:

A password change in PostgreSQL
Accidental removal of secret values

Warning

Changing or deleting the values of AKS secrets may break kdb Insights Enterprise on Azure. If you are not sure about the correct values and/or secrets to set, please reach out to KX Support team for assistance.

Prerequisites

To change or modify secrets on AKS, a command line tool like kubectl must be configured to communicate with the cluster. Before you make the modifications you will need the following:

Access to the Azure Portal
Access to the Kubernetes Cluster Overview
Access to Cloud Shell or Azure CLI
Kubectl CLI

Saving secrets

Depending on your company policy and needs, you may wish to save the different secret values to another secret store in case of corruption or accidental deletion.

Listing secrets

List all secrets across all namespaces:

 kubectl get secrets --all-namespaces

List secrets for a particular namespace:

kubectl get secrets --namespace "insights"

Decoding a secret

Secrets need to be formatted (in this example with JSONPath) and decoded, as they are base64 encoded. The namespace where the current secret resides also needs to be defined, or we can iterate over all namespaces with -A (short for --all-namespaces).

kubectl get secrets/kxi-acr-pull-secret --namespace "insights" --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode

This will output the content of the secret in a decoded and readable format.

To save the secret in base64 encoded format, just omit the decode command after the pipe to have the output still encoded:

kubectl get secrets/kxi-acr-pull-secret --namespace "insights" --output="jsonpath={.data.\.dockerconfigjson}"

Editing secrets

Since the kxi-acr-pull-secret mentioned as an example earlier is a Dockercfg secret that is used to authenticate against a Docker registry, we need to tell kubectl it is of type "docker-registry".

Changing ACR URL

kxi-acr-pull-secret's --docker-server attribute will change which ACR is used for pulling images. Currently changing the ACR to any other than what was the original URL at the time of deployment is not supported. Especially rook-ceph is affected negatively by this change.

kubectl edit --namespace "insights" secret docker-registry kxi-acr-pull-secret \
    --docker-username=<ACR_USERNAME> \
    --docker-password=<ACR_PASSWORD> \
    --docker-server=<ACR_ADDRESS> \
    --dry-run=client -o yaml | kubectl apply -f -

If the secret needs to be changed across multiple namespaces, it is possible to iterate over the command, for example:

kubectl get secret -A | grep kxi-acr-pull-secret | sed 's/ .*//' | while read -r ns ; do
  kubectl edit --namespace "${ns}" secret docker-registry kxi-acr-pull-secret \
    --docker-username=<ACR_USERNAME> \
    --docker-password=<ACR_PASSWORD> \
    --docker-server=<ACR_ADDRESS> \
    --dry-run=client -o yaml | kubectl apply -f -
done

Creating a secret

This section is applicable if the secret has been removed by accident.

To create a new secret, the namespace and type of secret needs to be defined.

kubectl create --namespace "insights" secret generic my-secret \
  --from-literal=username=username \
  --from-literal=password=topsecret

To recreate the above kxi-acr-pull-secret in the "insights" namespace:

kubectl create --namespace "insights" secret docker-registry kxi-acr-pull-secret \
    --docker-username=<ACR_USERNAME> \
    --docker-password=<ACR_PASSWORD> \
    --docker-server=<ACR_ADDRESS> \
    --dry-run=client -o yaml | kubectl apply -f -

Warning

As there usually isn't a reason to delete a secret on the kdb Insights Enterprise AKS, we have intentionally left the command out of the examples written here. KX advises using caution and recommends that you should be careful when deleting or modifying a secret.