Backup and Restore

This page explains how to back up DB Service data and how to restore from a backup.

Clustered deployments only

This page covers backup and restore for clustered (Kubernetes) deployments. All procedures use kubectl and assume tier data is held on persistent volume claims (PVCs), and that $NAMESPACE is set to the namespace you deployed into, as in the quickstart. Single-node (Docker Compose) deployments hold data on local storage, which changes both the procedures and the data-loss conditions described below.

Back up before schema changes

Always take a backup before modifying table schemas, running an initial import, or upgrading the DB Service. Schema changes and imports modify the HDB partition structure and cannot be automatically reversed.

Data layout

The DB Service stores data across the storage tiers below. Back up according to your recovery requirements:

Tier	Location	Back up?
RDB	In-memory	No — data is re-ingested from the RT stream after restart. Data received since the last EOI is at risk if the system restarts unexpectedly.
IDB	On-disk (PVC)	Yes, recommended — holds intraday write-downs, which are lost on restart if not backed up.
HDB	On-disk (PVC)	Yes, required — long-term historical data. Permanent loss if storage is corrupted or deleted.
Object storage	Cloud object store (S3, Azure Blob, GCS)	Yes, required — use your cloud provider's native snapshot or cross-region replication features. The kubectl procedures on this page do not apply to this tier.

For full protection, back up every on-disk tier: the IDB and HDB, plus any object storage tier you have configured.

Offline backup

An offline backup copies tier data while the system is stopped. It is the simplest and most consistent approach.

Stop after EOD completes

Stop the system only after an end-of-day (EOD) writedown has finished. Check SM logs for EOD complete, elapsed <time> before proceeding. This ensures all data has been flushed to disk and all tiers are consistent.

1. Stop the DB Service (scale down all deployments):

   kubectl scale deployment --all --replicas=0 --namespace "$NAMESPACE"
   

2. For each tier (IDB and HDB), create a backup pod that mounts the tier's PVC:

   backup-pod.yaml

   apiVersion: v1
   kind: Pod
   metadata:
     name: backup-pod
   spec:
     containers:
     - name: ubuntu
       image: ubuntu
       tty: true
       stdin: true
       volumeMounts:
         - mountPath: /data
           name: database-volume
     volumes:
       - name: database-volume
         persistentVolumeClaim:
           claimName: myasm-hdb   # Set to the PVC of the tier you want to back up
   

   kubectl apply -f backup-pod.yaml --namespace "$NAMESPACE"
   

   The pod must be in the same namespace as the PVC it mounts.

3. Create a compressed archive of the data directory. Using tar is required to preserve the symbolic links that SM uses internally:

   kubectl exec backup-pod --namespace "$NAMESPACE" -- sh -c "tar czf /data/backup.tar.gz /data/*"
   

4. Download the archive:

   kubectl cp backup-pod:/data/backup.tar.gz backup-hdb.tar.gz --namespace "$NAMESPACE"
   

5. Delete the backup pod and repeat steps 2–4 for each tier you want to back up (update claimName and the output filename).

6. Restart the DB Service:

   kubectl scale deployment --all --replicas=1 --namespace "$NAMESPACE"
   

Online backup (snapshots)

For systems that cannot tolerate downtime, the Storage Manager provides a snapshot API. A snapshot is a point-in-time copy of all tiers created using hard links. Taking a snapshot suspends EOI and EOD operations while it runs, ensuring no data changes mid-snapshot.

Send a POST request to the SM directly:

curl -X POST "http://$SM_HOST/snapshot"

The response returns the snapshot location for each tier:

[
  {
    "tier": "idb",
    "snapRoot": "/data/db/idb/snapshot/20230723160308643935166",
    "inventory": "/data/db/idb/snapshot/20230723160308643935166/inventory"
  },
  {
    "tier": "hdb",
    "snapRoot": "/data/db/hdb/snapshot/20230723160308643935166",
    "inventory": "/data/db/hdb/snapshot/20230723160308643935166/inventory"
  }
]

Copy the contents of each snapRoot to a safe storage location after the snapshot completes.

RT streaming log backup

The Reliable Transport (RT) streaming log records all ingest messages. Archiving the RT log allows you to replay ingest data in the event of a failure, recovering data that has not yet been written to disk.

RT log archiving is configured in the RT assembly settings. For details, see the Insights RT configuration reference.

Restore

Restoring from an offline backup

1. Scale down the DB Service:

   kubectl scale deployment --all --replicas=0 --namespace "$NAMESPACE"
   

2. For each tier, create a restore pod mounting the tier's PVC and extract the backup archive into it:

   kubectl exec restore-pod --namespace "$NAMESPACE" -- sh -c "tar xzf /data/backup-hdb.tar.gz -C /"
   

3. Restart the DB Service:

   kubectl scale deployment --all --replicas=1 --namespace "$NAMESPACE"
   

Restoring from a snapshot

Copy the snapRoot directory contents back to the data directory of the corresponding tier, then restart SM. SM picks up the restored data on startup.

Next steps

• Delete data
• Storage tiers
• Troubleshooting