Skip to content

Backup and Restore

This page provides instructions on how to backup and restore data using the CLI.

The kxi backup command, in the kdb Insights Command Line Interface, allows you to backup and restore data stored in kdb Insights Enterprise. It supports backup to any kind of object stores, including those managed by K8up.

The following data repositories are backed up as part of this process:

  • Historical database (HDB)
  • Intraday database (IDB)
  • Packages repository
  • Postgres databases for Authentication and Entitlements. The inclusion of these databases in backup and restore is optional.

Restoring to another cluster is not supported

If you need to restore to another cluster please contact KX Product Support.

This page provides information on the following:

Prerequisites

Before taking a backup, the following need to be in place:

  • The latest version of kdb Insights CLI, installed and configured to point to your kdb Insights Enterprise deployment.
  • kubectl access to the kdb Insights Enterprise cluster.
  • KX Management Service 0.2.8 with Insights On K8s 0.3.5 module package. To ensure you have the latest versions, run an upgrade using cli install upgrade, which prompts you to upgrade these if necessary.

  • Depending on the backup storage location, ensure the following resources are accessible from your kdb Insights Enterprise deployment:

    • For Google Cloud Storage: a GCS bucket.
    • For Microsoft Azure Storage: an Azure storage account.
    • For AWS S3: An S3 protocol compatible endpoint.

Instructions on accessing these cloud storage resources is outside the scope of this documentation. Refer to the relevant third party documentation.

Backup

The kxi backup command is used to initialize and run the backup and restore. It includes several related commands, which are discussed in this page and also covered in the reference guide:

Command Description
kxi backup init-backup Initialize the cloud storage details
kxi backup create-backup Run a backup
kxi backup watch Check the progress of a backup or restore
kxi backup populate-snapshots Collect the completed backups in a specific bucket/blob container
kxi backup restore-pvc Restore the HDB, IDB and packages repository
kxi backup restore-postgres Restore the Postgres databases

Initialization

The kdb Insights CLI needs the cloud storage details to be initialized.

This initialization only needs to be done before the first backup is taken, when the credentials change or when the version changes.

  1. Run the kxi backup init-backup command as follows:

    kxi backup init-backup -n <NAMESPACE> --provider <PROVIDER> --bucket-name <BUCKETNAME> --restic-pw <RESTICPW> --secret-name <SECRETNAME> --credentials <CREDENTIALS>

    Where:

    Arguments Required Details
    NAMESPACE Yes The namespace name where kdb Insights Enterprise runs.
    BUCKETNAME Yes The name of the target object store container where data are stored.
    PROVIDER Yes The type of cloud provider: "s3", "azure", or "gs".
    RESTICPW Yes A user defined encryption password to protect the backup storage. Take a note of the password you choose, as you need it to access the data in the backup.
    SECRETNAME No The name of the secret where the provider, bucket name, and storage credentials are stored. It has a default value of "backup-credentials".
    CREDENTIALS Yes The path to the credentials in a JSON file matching the provider. Click on a tab below to see the credentials for each of the cloud providers.

    The credentials are as follows

    {
"azureStgAccName": "value",
"azureStgAccKey": "value"
}
    Key Required Description
    azureStgAccName Yes The name of the Azure Storage Account.
    azureStgAccKey Yes The access key to the same Storage Account.

    The credentials are as follows

    {
"accessKeyID": "value",
"accessKey": "value",
"endpoint": "value"
}
    Key Required Description
    accessKeyID Yes The S3 access key ID. Usually 20 char long.
    accessKey Yes The S3 access key. Usually 40 char long.
    endpoint Yes The URL of the S3 provider. For AWS S3 use: https://s3.__region__.amazonaws.com and for S3 compatible services use IP or DNS address.

    AWS hosted bucket region

    If you are unable to access the AWS hosted bucket with administrator permissions to identify its region, use the following command to retrieve it:

    curl -sI https://<bucketname>.s3.amazonaws.com | grep bucket-region

    The credentials are as follows

    {
"type": "value",
"project_id": "value",
"private_key_id": "value",
"private_key": "value",
"client_email": "value",
"client_id": "value",
"auth_uri": "value",
"token_uri": "value",
"auth_provider_x509_cert_url": "value",
"client_x509_cert_url": "value",
"universe_domain": "value"
}
    Arguments Required Details
    type Yes The google cloud service account type.
    project_id Yes The google cloud Project ID.
    private_key_id Yes The google cloud private key ID.
    private_key Yes The google cloud private key.
    client_email Yes The google cloud client email.
    client_id Yes The google cloud client ID.
    auth_uri Yes The google cloud authorization URI.
    token_uri Yes The google cloud token URI.
    auth_provider_x509_cert_url Yes The google cloud auth provider X.509 cert URL.
    client_x509_cert_url Yes The google cloud client X.509 cert URL.
    universe_domain Yes The google cloud universe domain.

    Google Cloud only provides short lived tokens

    A service account needs to be created and exported from keycloak, refer to managing service accounts for details on creating a service account. The export is a JSON containing the values seen in the credential sections.

  2. When the initialization is successfully complete, the following messages are displayed:

    Task action: [JobStart        ]
Task action: [JobStepStart    ] - Helm repo for k8up addition step started
Task action: [JobStepEnd      ] - Helm repo for k8up addition step ended
Task action: [JobStepStart    ] - K8up Helm chart installation step started
Task action: [JobStepEnd      ] - K8up Helm chart installation step ended
Task action: [JobStepStart    ] - K8up CRDs installation step started
Task action: [JobStepEnd      ] - K8up CRDs installation step ended
Task action: [JobStepStart    ] - K8s Secrets creation step started
Task action: [JobStepEnd      ] - K8s Secrets creation step ended
Task action: [JobStepStart    ] - Populate-snapshots based on Restic repo step started
Task action: [JobStepEnd      ] - Populate-snapshots based on Restic repo step ended
Task action: [JobEnd          ]

    Executing the initialization command above performs the following actions:

    • Installs K8up operator with the respective CRDs.
    • Creates secrets to store storage backend related credentials and provider type using a default secret name, or a specified one.
    • Checks if the credentials are configured properly.
    • Checks for the presence of an existing Restic repository.
    • If data is detected, it populates K8up Snapshot Custom Resources, this helps to identify backups taken on foreign K8s clusters.
    • If it detects the credentials are correct and if the bucket does not exist with the configured name, it creates it.

    Changing credentials

    If you change the credentials in the same secret, remove the already existing secret from Kubernetes namespace before you run this initialization again.

Multiple backups

To configure and run multiple backups use a different combination of values for the provider, bucket name and storage credentials.

A different secret name, which provides the unique combination of values, can be used for each set of backup details you require. All tasks whose manifests share the same secret name will share the same backup details.

If the secret name is blank, in the manifests, a default name (backup-credentials) will be used.

Start the backup

Before you start the backup you must teardown the following:

  1. All databases that are running.

    Teardown but not clean-up

    • From the web interface make sure you do not check the clean-up resources option as that deletes all the RT logs as well.

    • From the CLI make sure you do not remove persistent data as that deletes all the RT logs as well.

  2. Any pipelines that are ingesting data into these databases.

    Teardown but do not clean-up

    From the web interface make sure you do not check the clean-up resources option as that deletes all pipeline checkpoints and user states as well.

    Ensuring continuous operation of publishers during database downtime

    Publishers, or feedhandlers, sending data to any of the databases that use an RT language interface can remain running as long as the local storage is sufficient to hold all the published messages while the database is offline.

    The publishers emit warning messages about the dropped connection if you keep them running.

To start the backup:

  1. Run the kxi backup create-backup command as follows:

    kxi backup create-backup -n <NAMESPACE> --bucket-name <BUCKETNAME> --backup-name BACKUPNAME --secret-name <SECRETNAME> --backup-postgres
    Arguments Required Details
    BACKUPNAME Yes This identifies the backup job, when checking the status.
    NAMESPACE Yes The namespace name where kdb Insights Enterprise runs.
    BUCKETNAME No The bucket name. If necessary, this can be used to override the bucket name set at initialization time.
    SECRETNAME No The name of the secret where the provider, bucket name, and storage credentials are read. This can be used to override the values set at initialization time if necessary.
    BACKUP_POSTGRES No A flag indicating whether to take a backup of the Postgres databases.

    Executing the backup command above performs the following actions:

    • Checks the installed K8up version and throws a warning if it is not installed, or a non-compatible version is present.
    • Annotates ReadWriteOnce volumes, as those are not in scope.
    • If --backup-postgres is set, the Postgres pod is annotated with the instructions to take a K8up application aware backup.
    • Creates a K8up Backup CR, which is picked up by the K8up controller and a backup of the ReadWriteMany volumes starts. If backupPostgres is set the Postgresql databases also get saved.
    • Shows the progress of the backup.

Annotations to ReadWriteOnce PVCs

The kdb Insights CLI annotates all ReadWriteOnce PVCs with the following k8up.io/backup=false before starting the backup to ensure that those PVCs are ignored.

Progress

When the backup has started, the following progress messages are displayed:

Deploying task backup create-backup
Task action: [JobStepStart    ] - K8up validation step started
Task action: [JobStepStart    ] - RWO volumes annotation step started
Task action: [JobStepEnd      ] - RWO volumes annotation step ended
Task action: [JobStepStart    ] - PostgreSQL annotation step started
Task action: [JobStepEnd      ] - PostgreSQL annotation step ended
Task action: [JobStepStart    ] - Backup CR creation step started
Task action: [JobStepEnd      ] - Backup CR creation step ended
Task action: [JobStepStart    ] - Started watching test12345
test12345: Succeeded, 00:00:32
backup-test12345-0-p8kpf: 100%|████████████████████████████████████████|, /data/test-package-idb
backup-test12345-1-7gxmb: 100%|███████████████████████████████████████|, /data/insights-packages-pvc

Ongoing backup progress can be checked with the following command.

kxi backup watch --task-type <TASK-TYPE> --task-name <TASK-NAME> -n <NAMESPACE>
Arguments Required Details
NAMESPACE Yes The namespace name where kdb Insights Enterprise runs.
TASK-TYPE Yes The task type to watch, backup or restore.
TASK-NAME Yes The user defined backup or restore name to watch.

On completion

When the backup is complete it is present in the K8up Snapshot list.

  1. We recommend that you check your Object store bucket or container, contains the following:

    /data
/index
/keys
/snapshots
config

  2. After a successful backup, execute kubectl delete backup <BACKUPNAME> -n <NAMESPACE> with the same arguments that kxi backup create-backup was initiated with. This deletes the kubernetes resources associated with the backup and ensures that there are no dangling resources.

Snapshots

To collect the completed backups in a specific bucket/blob container, call the kxi backup populate-snapshots command.

K8up populates the Snapshot K8up CRs after every backup job run, if the backups were run on a foreign cluster with help of the populate-snapshots task those can be retrieved and populated.

  1. Run the kxi backup populate-snapshots command:

    kxi backup populate-snapshots -n <NAMESPACE> --bucket-name <BUCKETNAME> --secret-name <SECRETNAME>
    Arguments Required Details
    NAMESPACE Yes The namespace name where kdb Insights Enterprise runs.
    BUCKETNAME No The bucket name. This can be used to override the values set at initialization time if required.
    SECRETNAME No The name of the secret where provider, bucket name, and storage credentials are read. This can be used to override the values set at initialization time if required.

  2. Run the following command to list the Snapshots, where NAMESPACE is the namespace where kdb Insights Enterprise runs.

    kubectl get snapshots -n <NAMESPACE>

    A list of completed backups in this bucket is returned, as illustrated in the example below:

    NAME       DATE TAKEN             PATHS                                                        REPOSITORY
058e0deb   2025-03-07T14:32:07Z   /data/dfx-test-package-hdb                                   azure:test:/
118978bb   2025-03-07T14:32:10Z   /insights-postgresql.sql                                     azure:test:/
488d4590   2025-03-07T14:25:35Z   /data/dfx-test-package-idb                                   azure:test:/

Restore

Restoration is done via the kxi backup restore-pvc command.

This can be used to restore:

Restoring to another cluster is not supported

If you need to restore to another cluster please contact KX Product Support.

HDB, IDB and packages

To restore the HDB, IDB and packages repository follow the steps below:

  1. Prepare to restore
  2. Run the restore
  3. Post restore steps

Restore HDB, IDB and packages from the same snapshot

All three of these data repositories must be restored from the same snapshot. When deploying the database the definition stored in the package must correspond to the schema used in the HDB and IDB.

Restoring to another cluster is not supported

If you need to restore to another cluster please contact Support.

Preparing to restore

Before you start the restore you need to take the following steps.

  1. Teardown the following:

    1. All running databases

      Teardown databases but do not clean-up

      • From the web interface make sure you do not check the clean-up resources option as that deletes all the RT logs as well.

      • From the CLI make sure you do not remove persistent data as that deletes all the RT logs as well.

    2. Any pipelines that are ingesting data into these databases

      Teardown pipelines but do not clean-up

      From the web interface make sure you do not check the clean-up resources option as that deletes all pipeline checkpoints and user states as well.

    Local storage for publishers when database is offline

    The publishers or feedhandlers, sending data to kdb Insights Enterprise that use an RT language interface can remain running as long as their local storage is sufficient to hold all the messages being published while the database is offline.

  2. Prepare the target volumes for the restore by creating a helper Ubuntu Pod with root access following these steps:

    Running containers as root could mean you receive warnings from security systems like Azure Defender.

    1. Save the manifest below to a .yaml file with the following values:

      Value Description
      NAMESPACE Namespace where kdb Insights Enterprise is deployed.
      PACKAGE The name of the package to which the database belongs. 
      apiVersion: v1
kind: Pod
metadata:
    name: helper-ubuntu
    namespace: <NAMESPACE>
spec:
    restartPolicy: Never    
    containers:
    - command: ["/bin/sh", "-c"]
      args:
      - "rm -rf /idb/* /hdb/*"
      image: ubuntu:22.04
      imagePullPolicy: IfNotPresent
      name: ubuntu
      resources: {}
      volumeMounts:
      - mountPath: /idb
        name: idb
      - mountPath: /hdb
        name: hdb
    securityContext:
        runAsUser: 0
        runAsNonRoot: false
    volumes:
    - name: idb
        persistentVolumeClaim:
            claimName: <PACKAGE>-idb
    - name: hdb
        persistentVolumeClaim:
            claimName: <PACKAGE>-hdb

    2. Apply this file with the following command:

      kubectl apply -f <filename.yaml>

Run the restore

After finishing all the necessary steps to prepare for the restore, you can proceed with starting the restore process.

  1. Run the kxi backup restore-pvc command as follows:

    kxi backup restore-pvc -n <NAMESPACE> --bucket-name <BUCKETNAME> --restore-name <RESTORENAME> --secret-name <SECRETNAME> --snapshot-id <SNAPSHOTID> --target-pvc <PVCNAME>
    Arguments Required Details
    NAMESPACE Yes The namespace name where kdb Insights Enterprise runs.
    BUCKETNAME No The bucket name. This can be used to override the values set at initialization time if required.
    RESTORENAME Yes The K8up Restore custom resource name you have chosen.
    SNAPSHOTID Yes The appropriate snapshot ID collected from the K8up snapshot list (e.g., 9513c1fr).
    SECRETNAME No The name of the secret where provider, bucket name, and storage credentials are read. This can be used to override the values set at initialization time if required.
    PVCNAME Yes Target PersistentVolumeClaim to restore (snapshot typically contains 'pvc').

  2. Progress is shown as follows:

    Deploying task backup restore-pvc
Task action: [JobStepStart    ] - K8up validation step started
Task action: [JobStepStart    ] - Restore CR creation step started
Task action: [JobStepEnd      ] - Restore CR creation step ended
Task action: [JobStepStart    ] - Started watching restore1.
restore1: Succeeded, 00:00:31
restore-restore1-w2zvr: 100%|██████████████████████████████████████████████████████████████████████|

  3. Ongoing restore progress can be checked with the following command.

    kxi backup watch --task-type <TASK-TYPE> --task-name <TASK-NAME> -n <NAMESPACE>
    Arguments Required Details
    NAMESPACE Yes The namespace name where kdb Insights Enterprise runs.
    TASK-TYPE Yes Task type to watch, backup or restore.
    TASK-NAME Yes User defined Backup or Restore name to watch.

Post restore steps

When the restore jobs are complete do the following:

  1. Deploy the restored databases and pipelines via the web interface or the CLI, and publishers that you previously torndown.

  2. Run a simple query to verify the restored data. You can do this using any of the querying methods available, including the web interface and REST.

Troubleshooting restore

  1. If the database is unable to start, use the following command to check the Storage Manager container logs. In rare cases the file system permissions are not correct. 

       kubectl logs <PACKAGE>-sm-0 --namespace <NAMESPACE>
    Key Description
    NAMESPACE Namespace where kdb Insights Enterprise is deployed
    PACKAGE The name of the package to which the database belongs.

    If this is the issue then initialization problems caused by permission issues are visible in the logs.

  2. Using the helper Ubuntu Pod with root access prepared earlier take the following actions to overwrite the permissions on the PVCs.

    Running containers as root could mean you receive warnings from security systems like Azure Defender.

    1. Save the manifest defined below to a .yaml file:

      apiVersion: v1
kind: Pod
metadata:
    name: helper-ubuntu
    namespace: <NAMESPACE>
spec:
    restartPolicy: Never    
    containers:
    - command: ["/bin/sh", "-c"]
      args:
      - "rm -rf /idb/* /hdb/*"
      image: ubuntu:22.04
      imagePullPolicy: IfNotPresent
      name: ubuntu
      resources: {}
      volumeMounts:
      - mountPath: /idb
        name: idb
      - mountPath: /hdb
        name: hdb
    securityContext:
        runAsUser: 0
        runAsNonRoot: false
    volumes:
    - name: idb
        persistentVolumeClaim:
            claimName: <PACKAGE>-idb
    - name: hdb
        persistentVolumeClaim:
            claimName: <PACKAGE>-hdb
      Key Description
      NAMESPACE The namespace where kdb Insights Enterprise is deployed.
      PACKAGE The name of the database to which the database belongs.

    2. Apply the file with the following command:

      kubectl apply -f <filename.yaml>

    3. Access the helper Ubuntu pod using the following command. Replace NAMESPACE with the name of the Kubernetes namespace where both kdb Insights Enterprise and the helper Ubuntu Pod are deployed:

      kubectl exec -it --namespace <NAMESPACE> helper-ubuntu -- /bin/bash

    4. Overwrite the permissions with the following commands:

      chown nobody:nogroup -R /idb
chmod 777 -R /idb
chown nobody:nogroup -R /hdb
chmod 777 -R /hdb

      Use Ctrl+D to quit from the container.

    5. Once the permissions are set on the affected PVCs, restart the database.

  3. Run the following command to check the SM container logs and ensure completed messages are logged rather than initialization issues:

    sh kubectl logs <PACKAGE>-sm-0 --namespace <NAMESPACE>

    Key Description
    NAMESPACE Namespace where kdb Insights Enterprise is deployed
    PACKAGE The name of the package to which the database belongs.

  4. Delete the helper Ubuntu Pod.

  5. After a successful restore, execute kubectl delete restore <RESTORENAME> -n <NAMESPACE> with the same arguments as the initial kxi backup restore-pvc command.

    This removes the Kubernetes resources created by the restore job, ensuring there are no leftover components that could interfere with future database deployments or teardowns.

Postgres database example

This section contains an example Postgres restoration, all use cases cannot be predicted, the restoration of the Postgres dump file is the main feature

We are providing example steps for Postgres level restoration only for the same deployment on the same version of Postgres and kdb Insights Enterprise.

To restore the Postgres, Keycloak and/or Entitlements, databases, follow the steps below:

  1. Run the following command:

    kxi backup restore-postgres -n <NAMESPACE> --bucket-name <BUCKETNAME> --secret-name <SECRETNAME> --snapshot-id <SNAPSHOTID>
    Arguments Required Details
    NAMESPACE Yes The namespace name where kdb Insights Enterprise runs.
    BUCKETNAME No The bucket name *
    SNAPSHOTID Yes The appropriate snapshot ID collected from the K8up snapshot list (e.g., 9513c1fr). This can be used to override the bucket name set at initialization time.
    SECRETNAME No The name of the secret where provider, bucket name, and storage credentials will be read. This can be used to override the bucket name set at initialization time.

    This restores the dump files and copies them to the /tmp directory of the insights-postgresql-0 pod with a name: /tmp/insights-postgresql-<SNAPSHOTID>.sql.

  2. Run the following command to set the number of replicas to 0 for the Keycloak StatefulSet to prevent modifications to the database while it is being restored:

    kubectl scale statefulset insights-keycloak --replicas=0 -n <NAMESPACE>
kubectl scale deployment insights-kxi-ent-srv --replicas=0 -n <NAMESPACE>

  3. Open a shell to insights-postgresql-0 pod:

    kubectl exec -it insights-postgresql-0 -n <NAMESPACE> -- sh

  4. Inside the pod shell, navigate to /tmp and make sure insights-postgresql-<SNAPSHOTID>.sql file exists:

    $ cd tmp
$ ls

  5. Extract from the cluster dump for the desired database, replacing with the appropriate value:

    start_line=$(grep -n "connect.*bitnami_keycloak" insights-postgresql-<SNAPSHOTID>.sql | cut -d ":" -f 1)
sed -n "${start_line},\${p;/PostgreSQL database dump complete/q}" insights-postgresql-<SNAPSHOTID>.sql > postgres_keycloak.sql

    start_line=$(grep -n "connect.*insights-kxi-ent" insights-postgresql-<SNAPSHOTID>.sql | cut -d ":" -f 1)
sed -n "${start_line},\${p;/PostgreSQL database dump complete/q}" insights-postgresql-<SNAPSHOTID>.sql > postgres_kxi_ent.sql

  6. Remaining in the /tmp directory, restore the desired database:

    PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "drop database bitnami_keycloak;" && \
PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "create database bitnami_keycloak;" && \
PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "grant all privileges on database bitnami_keycloak to bn_keycloak;" && \
PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "alter database bitnami_keycloak owner to bn_keycloak;" && \
PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres bitnami_keycloak < postgres_keycloak.sql;

    PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "drop database \"insights-kxi-ent\";" && \
PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "create database \"insights-kxi-ent\";" && \
PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "grant all privileges on database \"insights-kxi-ent\" to bn_keycloak;" && \
PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "alter database \"insights-kxi-ent\" owner to bn_keycloak;" && \
PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres "insights-kxi-ent" < postgres_kxi_ent.sql;

  7. Detach from the insights-postgresql-0 shell pod:

    $ exit

  8. Scale the number of Keycloak replicas back to 1.

    kubectl scale statefulset insights-keycloak --replicas=1 -n <NAMESPACE>
kubectl scale deployment insights-kxi-ent-srv --replicas=1 -n <NAMESPACE>

  9. Run a simple query to verify the postgres database is running as expected. You can do this using any of the querying methods available, including:

    By logging in and querying the data, you are confirming that these databases has been successfully restored.