Backup and Restore
This page provides instructions on how to backup and restore data using the CLI.
The kxi backup
command, in the kdb Insights Command Line Interface, allows you to backup and restore data stored in kdb Insights Enterprise. It supports backup to any kind of object stores, including those managed by K8up.
The following data repositories are backed up as part of this process:
- Historical database (HDB)
- Intraday database (IDB)
- Packages repository
- Postgres databases for Authentication and Entitlements. The inclusion of these databases in backup and restore is optional.
Restoring to another cluster is not supported
If you need to restore to another cluster please contact KX Product Support.
This page provides information on the following:
- The prerequisite steps to take before performing a backup
- Initializing a backup
- Starting the backup
- Checking the progress of the backup
- Steps to take when the backup completes
- Creating and verifying snapshots of your backup
- Restoring HDB, IDB and packages
- Example postgres restoration
Prerequisites
Before taking a backup, the following need to be in place:
- The latest version of kdb Insights CLI, installed and configured to point to your kdb Insights Enterprise deployment.
kubectl
access to the kdb Insights Enterprise cluster.- KX Management Service 0.2.8 with Insights On K8s 0.3.5 module package. To ensure you have the latest versions, run an upgrade using
cli install upgrade
, which prompts you to upgrade these if necessary. -
Depending on the backup storage location, ensure the following resources are accessible from your kdb Insights Enterprise deployment:
- For Google Cloud Storage: a GCS bucket.
- For Microsoft Azure Storage: an Azure storage account.
- For AWS S3: An S3 protocol compatible endpoint.
Instructions on accessing these cloud storage resources is outside the scope of this documentation. Refer to the relevant third party documentation.
Backup
The kxi backup
command is used to initialize and run the backup and restore. It includes several related commands, which are discussed in this page and also covered in the reference guide:
Command | Description |
---|---|
kxi backup init-backup |
Initialize the cloud storage details |
kxi backup create-backup |
Run a backup |
kxi backup watch |
Check the progress of a backup or restore |
kxi backup populate-snapshots |
Collect the completed backups in a specific bucket/blob container |
kxi backup restore-pvc |
Restore the HDB, IDB and packages repository |
kxi backup restore-postgres |
Restore the Postgres databases |
Initialization
The kdb Insights CLI needs the cloud storage details to be initialized.
This initialization only needs to be done before the first backup is taken, when the credentials change or when the version changes.
-
Run the
kxi backup init-backup
command as follows:kxi backup init-backup -n <NAMESPACE> --provider <PROVIDER> --bucket-name <BUCKETNAME> --restic-pw <RESTICPW> --secret-name <SECRETNAME> --credentials <CREDENTIALS>
Where:
Arguments Required Details NAMESPACE
Yes The namespace name where kdb Insights Enterprise runs. BUCKETNAME
Yes The name of the target object store container where data are stored. PROVIDER
Yes The type of cloud provider: "s3", "azure", or "gs". RESTICPW
Yes A user defined encryption password to protect the backup storage. Take a note of the password you choose, as you need it to access the data in the backup. SECRETNAME
No The name of the secret where the provider, bucket name, and storage credentials are stored. It has a default value of "backup-credentials". CREDENTIALS
Yes The path to the credentials in a JSON file matching the provider. Click on a tab below to see the credentials for each of the cloud providers. The credentials are as follows
{ "azureStgAccName": "value", "azureStgAccKey": "value" }
Key Required Description azureStgAccName
Yes The name of the Azure Storage Account. azureStgAccKey
Yes The access key to the same Storage Account. The credentials are as follows
{ "accessKeyID": "value", "accessKey": "value", "endpoint": "value" }
Key Required Description accessKeyID
Yes The S3 access key ID. Usually 20 char long. accessKey
Yes The S3 access key. Usually 40 char long. endpoint
Yes The URL of the S3 provider. For AWS S3 use: https://s3.__region__.amazonaws.com
and for S3 compatible services use IP or DNS address.AWS hosted bucket region
If you are unable to access the AWS hosted bucket with administrator permissions to identify its region, use the following command to retrieve it:
curl -sI https://<bucketname>.s3.amazonaws.com | grep bucket-region
The credentials are as follows
{ "type": "value", "project_id": "value", "private_key_id": "value", "private_key": "value", "client_email": "value", "client_id": "value", "auth_uri": "value", "token_uri": "value", "auth_provider_x509_cert_url": "value", "client_x509_cert_url": "value", "universe_domain": "value" }
Arguments Required Details type
Yes The google cloud service account type. project_id
Yes The google cloud Project ID. private_key_id
Yes The google cloud private key ID. private_key
Yes The google cloud private key. client_email
Yes The google cloud client email. client_id
Yes The google cloud client ID. auth_uri
Yes The google cloud authorization URI. token_uri
Yes The google cloud token URI. auth_provider_x509_cert_url
Yes The google cloud auth provider X.509 cert URL. client_x509_cert_url
Yes The google cloud client X.509 cert URL. universe_domain
Yes The google cloud universe domain. Google Cloud only provides short lived tokens
A service account needs to be created and exported from keycloak, refer to managing service accounts for details on creating a service account. The export is a JSON containing the values seen in the credential sections.
-
When the initialization is successfully complete, the following messages are displayed:
Task action: [JobStart ] Task action: [JobStepStart ] - Helm repo for k8up addition step started Task action: [JobStepEnd ] - Helm repo for k8up addition step ended Task action: [JobStepStart ] - K8up Helm chart installation step started Task action: [JobStepEnd ] - K8up Helm chart installation step ended Task action: [JobStepStart ] - K8up CRDs installation step started Task action: [JobStepEnd ] - K8up CRDs installation step ended Task action: [JobStepStart ] - K8s Secrets creation step started Task action: [JobStepEnd ] - K8s Secrets creation step ended Task action: [JobStepStart ] - Populate-snapshots based on Restic repo step started Task action: [JobStepEnd ] - Populate-snapshots based on Restic repo step ended Task action: [JobEnd ]
Executing the initialization command above performs the following actions:
- Installs K8up operator with the respective CRDs.
- Creates secrets to store storage backend related credentials and provider type using a default secret name, or a specified one.
- Checks if the credentials are configured properly.
- Checks for the presence of an existing Restic repository.
- If data is detected, it populates K8up Snapshot Custom Resources, this helps to identify backups taken on foreign K8s clusters.
- If it detects the credentials are correct and if the bucket does not exist with the configured name, it creates it.
Changing credentials
If you change the credentials in the same secret, remove the already existing secret from Kubernetes namespace before you run this initialization again.
Multiple backups
To configure and run multiple backups use a different combination of values for the provider, bucket name and storage credentials.
A different secret name, which provides the unique combination of values, can be used for each set of backup details you require. All tasks whose manifests share the same secret name will share the same backup details.
If the secret name is blank, in the manifests, a default name (backup-credentials) will be used.
Start the backup
Before you start the backup you must teardown the following:
-
All databases that are running.
Teardown but not clean-up
-
From the web interface make sure you do not check the clean-up resources option as that deletes all the RT logs as well.
-
From the CLI make sure you do not remove persistent data as that deletes all the RT logs as well.
-
-
Any pipelines that are ingesting data into these databases.
Teardown but do not clean-up
From the web interface make sure you do not check the clean-up resources option as that deletes all pipeline checkpoints and user states as well.
Ensuring continuous operation of publishers during database downtime
Publishers, or feedhandlers, sending data to any of the databases that use an RT language interface can remain running as long as the local storage is sufficient to hold all the published messages while the database is offline.
The publishers emit warning messages about the dropped connection if you keep them running.
To start the backup:
-
Run the
kxi backup create-backup
command as follows:kxi backup create-backup -n <NAMESPACE> --bucket-name <BUCKETNAME> --backup-name BACKUPNAME --secret-name <SECRETNAME> --backup-postgres
Arguments Required Details BACKUPNAME
Yes This identifies the backup job, when checking the status. NAMESPACE
Yes The namespace name where kdb Insights Enterprise runs. BUCKETNAME
No The bucket name. If necessary, this can be used to override the bucket name set at initialization time. SECRETNAME
No The name of the secret where the provider, bucket name, and storage credentials are read. This can be used to override the values set at initialization time if necessary. BACKUP_POSTGRES
No A flag indicating whether to take a backup of the Postgres databases. Executing the backup command above performs the following actions:
- Checks the installed K8up version and throws a warning if it is not installed, or a non-compatible version is present.
- Annotates ReadWriteOnce volumes, as those are not in scope.
- If
--backup-postgres
is set, the Postgres pod is annotated with the instructions to take a K8up application aware backup. - Creates a K8up Backup CR, which is picked up by the K8up controller and a backup of the ReadWriteMany volumes starts. If
backupPostgres
is set the Postgresql databases also get saved. - Shows the progress of the backup.
Annotations to ReadWriteOnce PVCs
The kdb Insights CLI annotates all ReadWriteOnce PVCs with the following k8up.io/backup=false
before starting the backup to ensure that those PVCs are ignored.
Progress
When the backup has started, the following progress messages are displayed:
Deploying task backup create-backup
Task action: [JobStepStart ] - K8up validation step started
Task action: [JobStepStart ] - RWO volumes annotation step started
Task action: [JobStepEnd ] - RWO volumes annotation step ended
Task action: [JobStepStart ] - PostgreSQL annotation step started
Task action: [JobStepEnd ] - PostgreSQL annotation step ended
Task action: [JobStepStart ] - Backup CR creation step started
Task action: [JobStepEnd ] - Backup CR creation step ended
Task action: [JobStepStart ] - Started watching test12345
test12345: Succeeded, 00:00:32
backup-test12345-0-p8kpf: 100%|████████████████████████████████████████|, /data/test-package-idb
backup-test12345-1-7gxmb: 100%|███████████████████████████████████████|, /data/insights-packages-pvc
Ongoing backup progress can be checked with the following command.
kxi backup watch --task-type <TASK-TYPE> --task-name <TASK-NAME> -n <NAMESPACE>
Arguments | Required | Details |
---|---|---|
NAMESPACE |
Yes | The namespace name where kdb Insights Enterprise runs. |
TASK-TYPE |
Yes | The task type to watch, backup or restore . |
TASK-NAME |
Yes | The user defined backup or restore name to watch. |
On completion
When the backup is complete it is present in the K8up Snapshot list.
-
We recommend that you check your Object store bucket or container, contains the following:
/data /index /keys /snapshots config
-
After a successful backup, execute
kubectl delete backup <BACKUPNAME> -n <NAMESPACE>
with the same arguments thatkxi backup create-backup
was initiated with. This deletes the kubernetes resources associated with the backup and ensures that there are no dangling resources.
Snapshots
To collect the completed backups in a specific bucket/blob container, call the kxi backup populate-snapshots
command.
K8up populates the Snapshot K8up CRs after every backup job run, if the backups were run on a foreign cluster with help of the populate-snapshots
task those can be retrieved and populated.
-
Run the
kxi backup populate-snapshots
command:kxi backup populate-snapshots -n <NAMESPACE> --bucket-name <BUCKETNAME> --secret-name <SECRETNAME>
Arguments Required Details NAMESPACE
Yes The namespace name where kdb Insights Enterprise runs. BUCKETNAME
No The bucket name. This can be used to override the values set at initialization time if required. SECRETNAME
No The name of the secret where provider, bucket name, and storage credentials are read. This can be used to override the values set at initialization time if required. -
Run the following command to list the Snapshots, where
NAMESPACE
is the namespace where kdb Insights Enterprise runs.kubectl get snapshots -n <NAMESPACE>
A list of completed backups in this bucket is returned, as illustrated in the example below:
NAME DATE TAKEN PATHS REPOSITORY 058e0deb 2025-03-07T14:32:07Z /data/dfx-test-package-hdb azure:test:/ 118978bb 2025-03-07T14:32:10Z /insights-postgresql.sql azure:test:/ 488d4590 2025-03-07T14:25:35Z /data/dfx-test-package-idb azure:test:/
Restore
Restoration is done via the kxi backup restore-pvc
command.
This can be used to restore:
- HDB, IDB and packages
- Postgres databases, if they have been backed up. Refer to Postgres database example for details.
Restoring to another cluster is not supported
If you need to restore to another cluster please contact KX Product Support.
HDB, IDB and packages
To restore the HDB, IDB and packages repository follow the steps below:
Restore HDB, IDB and packages from the same snapshot
All three of these data repositories must be restored from the same snapshot. When deploying the database the definition stored in the package must correspond to the schema used in the HDB and IDB.
Restoring to another cluster is not supported
If you need to restore to another cluster please contact Support.
Preparing to restore
Before you start the restore you need to take the following steps.
-
Teardown the following:
-
All running databases
Teardown databases but do not clean-up
-
From the web interface make sure you do not check the clean-up resources option as that deletes all the RT logs as well.
-
From the CLI make sure you do not remove persistent data as that deletes all the RT logs as well.
-
-
Any pipelines that are ingesting data into these databases
Teardown pipelines but do not clean-up
From the web interface make sure you do not check the clean-up resources option as that deletes all pipeline checkpoints and user states as well.
Local storage for publishers when database is offline
The publishers or feedhandlers, sending data to kdb Insights Enterprise that use an RT language interface can remain running as long as their local storage is sufficient to hold all the messages being published while the database is offline.
-
-
Prepare the target volumes for the restore by creating a helper Ubuntu Pod with root access following these steps:
Running containers as root could mean you receive warnings from security systems like Azure Defender.
-
Save the manifest below to a
.yaml
file with the following values:Value Description NAMESPACE
Namespace where kdb Insights Enterprise is deployed. PACKAGE
The name of the package to which the database belongs. apiVersion: v1 kind: Pod metadata: name: helper-ubuntu namespace: <NAMESPACE> spec: restartPolicy: Never containers: - command: ["/bin/sh", "-c"] args: - "rm -rf /idb/* /hdb/*" image: ubuntu:22.04 imagePullPolicy: IfNotPresent name: ubuntu resources: {} volumeMounts: - mountPath: /idb name: idb - mountPath: /hdb name: hdb securityContext: runAsUser: 0 runAsNonRoot: false volumes: - name: idb persistentVolumeClaim: claimName: <PACKAGE>-idb - name: hdb persistentVolumeClaim: claimName: <PACKAGE>-hdb
-
Apply this file with the following command:
kubectl apply -f <filename.yaml>
-
Run the restore
After finishing all the necessary steps to prepare for the restore, you can proceed with starting the restore process.
-
Run the
kxi backup restore-pvc
command as follows:kxi backup restore-pvc -n <NAMESPACE> --bucket-name <BUCKETNAME> --restore-name <RESTORENAME> --secret-name <SECRETNAME> --snapshot-id <SNAPSHOTID> --target-pvc <PVCNAME>
Arguments Required Details NAMESPACE
Yes The namespace name where kdb Insights Enterprise runs. BUCKETNAME
No The bucket name. This can be used to override the values set at initialization time if required. RESTORENAME
Yes The K8up Restore custom resource name you have chosen. SNAPSHOTID
Yes The appropriate snapshot ID collected from the K8up snapshot list (e.g., 9513c1fr). SECRETNAME
No The name of the secret where provider, bucket name, and storage credentials are read. This can be used to override the values set at initialization time if required. PVCNAME
Yes Target PersistentVolumeClaim to restore (snapshot typically contains 'pvc'). -
Progress is shown as follows:
Deploying task backup restore-pvc Task action: [JobStepStart ] - K8up validation step started Task action: [JobStepStart ] - Restore CR creation step started Task action: [JobStepEnd ] - Restore CR creation step ended Task action: [JobStepStart ] - Started watching restore1. restore1: Succeeded, 00:00:31 restore-restore1-w2zvr: 100%|██████████████████████████████████████████████████████████████████████|
-
Ongoing restore progress can be checked with the following command.
kxi backup watch --task-type <TASK-TYPE> --task-name <TASK-NAME> -n <NAMESPACE>
Arguments Required Details NAMESPACE
Yes The namespace name where kdb Insights Enterprise runs. TASK-TYPE
Yes Task type to watch, backup
orrestore
.TASK-NAME
Yes User defined Backup or Restore name to watch.
Post restore steps
When the restore jobs are complete do the following:
-
Deploy the restored databases and pipelines via the web interface or the CLI, and publishers that you previously torndown.
-
Run a simple query to verify the restored data. You can do this using any of the querying methods available, including the web interface and REST.
Troubleshooting restore
-
If the database is unable to start, use the following command to check the Storage Manager container logs. In rare cases the file system permissions are not correct.
kubectl logs <PACKAGE>-sm-0 --namespace <NAMESPACE>
Key Description NAMESPACE
Namespace where kdb Insights Enterprise is deployed PACKAGE
The name of the package to which the database belongs. If this is the issue then initialization problems caused by permission issues are visible in the logs.
-
Using the helper Ubuntu Pod with root access prepared earlier take the following actions to overwrite the permissions on the PVCs.
Running containers as root could mean you receive warnings from security systems like Azure Defender.
-
Save the manifest defined below to a
.yaml
file:apiVersion: v1 kind: Pod metadata: name: helper-ubuntu namespace: <NAMESPACE> spec: restartPolicy: Never containers: - command: ["/bin/sh", "-c"] args: - "rm -rf /idb/* /hdb/*" image: ubuntu:22.04 imagePullPolicy: IfNotPresent name: ubuntu resources: {} volumeMounts: - mountPath: /idb name: idb - mountPath: /hdb name: hdb securityContext: runAsUser: 0 runAsNonRoot: false volumes: - name: idb persistentVolumeClaim: claimName: <PACKAGE>-idb - name: hdb persistentVolumeClaim: claimName: <PACKAGE>-hdb
Key Description NAMESPACE
The namespace where kdb Insights Enterprise is deployed. PACKAGE
The name of the database to which the database belongs. -
Apply the file with the following command:
kubectl apply -f <filename.yaml>
-
Access the helper Ubuntu pod using the following command. Replace
NAMESPACE
with the name of the Kubernetes namespace where both kdb Insights Enterprise and the helper Ubuntu Pod are deployed:kubectl exec -it --namespace <NAMESPACE> helper-ubuntu -- /bin/bash
-
Overwrite the permissions with the following commands:
chown nobody:nogroup -R /idb chmod 777 -R /idb chown nobody:nogroup -R /hdb chmod 777 -R /hdb
Use Ctrl+D to quit from the container.
-
Once the permissions are set on the affected PVCs, restart the database.
-
-
Run the following command to check the SM container logs and ensure completed messages are logged rather than initialization issues:
sh kubectl logs <PACKAGE>-sm-0 --namespace <NAMESPACE>
Key Description NAMESPACE
Namespace where kdb Insights Enterprise is deployed PACKAGE
The name of the package to which the database belongs. -
Delete the helper Ubuntu Pod.
-
After a successful restore, execute
kubectl delete restore <RESTORENAME> -n <NAMESPACE>
with the same arguments as the initialkxi backup restore-pvc
command.This removes the Kubernetes resources created by the restore job, ensuring there are no leftover components that could interfere with future database deployments or teardowns.
Postgres database example
This section contains an example Postgres restoration, all use cases cannot be predicted, the restoration of the Postgres dump file is the main feature
We are providing example steps for Postgres level restoration only for the same deployment on the same version of Postgres and kdb Insights Enterprise.
To restore the Postgres, Keycloak and/or Entitlements, databases, follow the steps below:
-
Run the following command:
kxi backup restore-postgres -n <NAMESPACE> --bucket-name <BUCKETNAME> --secret-name <SECRETNAME> --snapshot-id <SNAPSHOTID>
Arguments Required Details NAMESPACE
Yes The namespace name where kdb Insights Enterprise runs. BUCKETNAME
No The bucket name * SNAPSHOTID
Yes The appropriate snapshot ID collected from the K8up snapshot list (e.g., 9513c1fr). This can be used to override the bucket name set at initialization time. SECRETNAME
No The name of the secret where provider, bucket name, and storage credentials will be read. This can be used to override the bucket name set at initialization time. This restores the dump files and copies them to the
/tmp
directory of theinsights-postgresql-0
pod with a name:/tmp/insights-postgresql-<SNAPSHOTID>.sql
. -
Run the following command to set the number of replicas to 0 for the Keycloak StatefulSet to prevent modifications to the database while it is being restored:
kubectl scale statefulset insights-keycloak --replicas=0 -n <NAMESPACE> kubectl scale deployment insights-kxi-ent-srv --replicas=0 -n <NAMESPACE>
-
Open a shell to
insights-postgresql-0
pod:kubectl exec -it insights-postgresql-0 -n <NAMESPACE> -- sh
-
Inside the pod shell, navigate to
/tmp
and make sureinsights-postgresql-<SNAPSHOTID>.sql
file exists:$ cd tmp $ ls
-
Extract from the cluster dump for the desired database, replacing
with the appropriate value: start_line=$(grep -n "connect.*bitnami_keycloak" insights-postgresql-<SNAPSHOTID>.sql | cut -d ":" -f 1) sed -n "${start_line},\${p;/PostgreSQL database dump complete/q}" insights-postgresql-<SNAPSHOTID>.sql > postgres_keycloak.sql
start_line=$(grep -n "connect.*insights-kxi-ent" insights-postgresql-<SNAPSHOTID>.sql | cut -d ":" -f 1) sed -n "${start_line},\${p;/PostgreSQL database dump complete/q}" insights-postgresql-<SNAPSHOTID>.sql > postgres_kxi_ent.sql
-
Remaining in the /tmp directory, restore the desired database:
PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "drop database bitnami_keycloak;" && \ PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "create database bitnami_keycloak;" && \ PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "grant all privileges on database bitnami_keycloak to bn_keycloak;" && \ PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "alter database bitnami_keycloak owner to bn_keycloak;" && \ PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres bitnami_keycloak < postgres_keycloak.sql;
PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "drop database \"insights-kxi-ent\";" && \ PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "create database \"insights-kxi-ent\";" && \ PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "grant all privileges on database \"insights-kxi-ent\" to bn_keycloak;" && \ PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "alter database \"insights-kxi-ent\" owner to bn_keycloak;" && \ PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres "insights-kxi-ent" < postgres_kxi_ent.sql;
-
Detach from the
insights-postgresql-0
shell pod:$ exit
-
Scale the number of Keycloak replicas back to 1.
kubectl scale statefulset insights-keycloak --replicas=1 -n <NAMESPACE> kubectl scale deployment insights-kxi-ent-srv --replicas=1 -n <NAMESPACE>
-
Run a simple query to verify the postgres database is running as expected. You can do this using any of the querying methods available, including:
By logging in and querying the data, you are confirming that these databases has been successfully restored.