Skip to content

Back up and restore a single-node system

This page explains what to back up on a single-node KX Sensors system that uses TP+, how to keep the backup set consistent so that log replay fills the gap after a restore, how to protect the TP+ logs and the etcd discovery registry, and how to recover after a disk failure.

Single-node systems only

This procedure applies only to single-node systems, where EMS streams run on TP+. It doesn't apply to multi-node systems, where EMS streams run on RT with Raft replication. Don't use it to back up or restore RT logs or a multi-node etcd cluster. For multi-node systems, see Recover your data.

Prerequisites

  • A single-node system that runs EMS streams on TP+. The procedure doesn't apply to multi-node systems.
  • A separate physical disk for tpLogBackupDir, and a separate disk or host for your backup sets and any off-host copies of emsLogDir.
  • The ability to stop and start the system with kxsctl. See Start and stop the system.

Overview

On a single-node system, there's no other node to fail over to, and no other copy of the data received since the last EOI. As well as the IDB and HDB, a restore depends on state that's kept outside the database:

  • The TP+ logs in emsLogDir, which hold all data received since the last EOI.
  • The per-process watermarks and checkpoints, which record the positions in the TP+ logs that processes resume from.
  • The etcd discovery registry, which holds node state, manifests, and the registry's view of the on-disk database.

For the general rules on backing up and recovering the IDB and HDB, such as preserving symlinks and paths, see Recover your data.

What to back up

The following tables group the locations to back up on a single-node system by how often you back them up.

Note

By default, wmDir and chkpDir are subdirectories of opDataDir, but you can set either of them to a location outside it in systemParams.yaml. Check the values in your deployment, and back up each directory wherever it's located. A backup of opDataDir alone doesn't include a relocated wmDir or chkpDir. Packages installed on top of KX Sensors can also override these parameters in their own systemParams.yaml, so check the effective values rather than only the defaults.

Note

Each location in the tables, apart from $KXS_ROOT itself and the etcd registry, is set by a system parameter in systemParams.yaml or, for KXS_LOG_DIR, by an environment variable in kxsenv. The defaults are under $KXS_ROOT, but any of them can be set elsewhere, in which case a backup of $KXS_ROOT doesn't include it. Check the values in your deployment, and back up each location wherever it's set. Where they're under $KXS_ROOT, exclude them from the daily backup of $KXS_ROOT, and back them up on their own schedule. A copy of those locations taken with $KXS_ROOT isn't from the same point in time as the rest of the backup set, as described in the next section, so don't restore from them from that copy.

Back up continuously

Location Contents
emsLogDir (default ${KXS_ROOT}/ems/${NODE}) TP+ logs, holding all data received since the last EOI. Protect them by using one of the options in Back up the TP+ logs.

Back up after each EOI

Back up these locations after EOI completes, all from the same point in time, as described in Keep the backup set consistent. On this page, the operational data directories means opDataDir, wmDir, and chkpDir together.

Location Contents
idbDir (default ${KXS_ROOT}/db/idb) Intraday data, a full copy of master data, the sym file, and the IdbStatus table, which records the TP+ log position that subscribers replay from after a restart.
opDataDir (default ${KXS_ROOT}/opdata_${NODE}/${INST}) Per-process operational data.
wmDir (default ${OPDATA}/wm) Watermarks, which record TP+ log replay positions, for example the position SM resumes from.
chkpDir (default ${OPDATA}/chkp) Checkpoints, which record the TP+ positions SPs replay from and TP+ deduplication positions.
etcd discovery registry Node state, manifests, database purview and vintage, HA state, and data loci. Back it up by using etcdctl snapshot save, as described in Back up the discovery registry.

Back up daily

Schedule these backups to run every day. A daily backup of $KXS_ROOT means that installation, upgrade, and configuration changes are protected without anyone having to remember to back them up after each change.

Location Contents When
$KXS_ROOT KX Sensors installation and configuration files. Exclude every other location on this page that's under $KXS_ROOT, including tpLogBackupDir, since each has its own schedule. At a set time each day.
hdbDir (default ${KXS_ROOT}/db/hdb) Historical database. If you use tiered storage, also back up each tier's directory, as set in hdbTiers.yaml. After EOD completes.
KXS_LOG_DIR (default ${KXS_ROOT}/logs) Log files. Outside EOD and log cutover.

Keep the backup set consistent

TP+ logs replay fills the gap between your backup and the present, which works only if the backup set is consistent. Follow these rules:

  • Capture the etcd registry, the operational data directories, and the IDB from the same point in time. Ideally, take a single file-system snapshot after EOI completes, and copy everything from it. If they're on different volumes, copy them in the order given in Choose a backup method, with the IDB last. The TP+ logs are protected separately, as described in Back up the TP+ logs.
  • Keep TP+ logs long enough. The stream's archiving parameters (retentionDuration, size, and diskUsage) in ems.yaml must keep the logs back to the oldest position in your most recent backup set, with a margin. If the logs that replay needs have been pruned, replay has a gap.
  • Restore the IDB and HDB from the same date to avoid database mismatches.

Note

Backing up wmDir or chkpDir more often than the IDB doesn't reduce data loss, even though checkpoints are written every few seconds. Watermarks and checkpoints record positions, not data. The data received since your last backup is in the TP+ logs, which is why you protect the TP+ logs continuously.

Back up the TP+ logs

The TP+ logs are the only copy of data received since the last EOI, so they need protection between your backups of the IDB. Choose one of the following options:

Option Data lost if the emsLogDir disk fails Who runs it
Option 1: Use TP+ log backups (recommended) At most about one flush interval (flushFreq, by default one second). TP+, as part of writing each stream.
Option 2: Copy the TP+ logs yourself Everything received since your last copy. You, with your own tools and schedule.

KX recommends that you enable TP+ log backups for every stream. TP+ then writes a second copy of every message to tpLogBackupDir and flushes it on the same cycle as the primary log. If the disk that holds emsLogDir fails, you lose at most about one flush interval of data.

When you use TP+ log backups:

  • Put tpLogBackupDir on a separate physical disk. The default, ${KXS_ROOT}/tpp_bkup, is usually on the same disk as emsLogDir, which defeats the purpose.
  • Monitor the backup. TP+ can disable the backup at startup and continue without it. See How TP+ checks the backup for what to alert on.

Option 2: Copy the TP+ logs yourself

If you'd rather protect the TP+ logs with your own tooling, you can leave TP+ log backups disabled and copy each stream's directory, <emsLogDir>/<stream>, with a tool such as rsync, while the system is running. Rolled log files never change, and only the newest file is still being written. Take the copy from a file-system snapshot so that the active file is consistent; a copy of the live active file can end partway through a message.

With this option:

  • The copy interval sets your data loss. If the disk that holds emsLogDir fails, you lose everything received since your last copy, so copy as often as your data-loss tolerance requires.
  • You're responsible for scheduling and monitoring the copy. KX Sensors doesn't know about the copy, so it doesn't report a copy that fails or falls behind.
  • Copy to a separate physical disk or host, so that a single disk failure doesn't lose both copies.
  • Add --checksum if each copy overwrites the previous one. TP+ rewrites small state files in place without changing their size, and without --checksum, rsync can skip them.

Back up the discovery registry

KX Sensors stores node state, manifests, the database purview and vintage, HA state, and data loci in etcd. Without the registry, the system can't start. The purview and vintage describe the on-disk IDB and HDB, so take the etcd backup as part of the same backup set as the IDB.

Back up the registry with an etcd snapshot (etcdctl snapshot save), which you can take while etcd is running. For the snapshot and restore commands, follow the etcd disaster recovery guide. Use the version of the etcd documentation that matches the etcd version installed with your deployment; see Upgrade etcd for how that version is set.

Warning

Don't back up etcd by copying its live data directory (ETCD_DATA_DIR) with rsync or cp. etcd writes to that directory continuously, so a file-by-file copy can be inconsistent. Use an etcd snapshot, as described in the etcd guide.

After each configuration change, also export the node's manifest with kxsctl manifest export so you have a readable copy outside etcd.

Choose a backup method

Use a combination of full and incremental backups:

  • Installation and configuration: a full copy of $KXS_ROOT every day, excluding the locations that you back up separately.
  • HDB: an incremental copy after EOD. Historical partitions don't change once written, so rsync -a transfers only the new partition and the files that changed. If you use row-level retention, see Back up and restore pruned data.
  • IDB, operational data directories, and etcd: a copy of all three after each EOI, from the same point in time.
  • Between EOIs: TP+ log backups (recommended), or your own frequent copies of emsLogDir. See Back up the TP+ logs.
  • Periodically: a full backup that you use to test your restore procedure.

Recover after a disk failure

  1. Stop the system by using kxsctl stop system.
  2. Replace or remount the failed storage at the same paths as before.
  3. If the installation or configuration was lost, restore $KXS_ROOT from backup.
  4. Restore the IDB, HDB, and operational data directories (opDataDir, wmDir, and chkpDir) from the same backup set, preserving symlinks and paths.

  5. If the etcd registry was lost, restore it from the snapshot in the same backup set, following the restore procedure in the etcd disaster recovery guide. For a KX Sensors system:

    • Stop the kxs-etcd and kxs-qetcd services before you restore, and start them again afterward.

    • Pass the restore command the member name, peer URL, cluster token, and data directory from etcd-conf.yml, so that the restored member matches the configuration etcd starts with:

      etcdctl snapshot restore <snapshot> \
  --name <name> \
  --initial-advertise-peer-urls <initial-advertise-peer-urls> \
  --initial-cluster <initial-cluster> \
  --initial-cluster-token <initial-cluster-token> \
  --data-dir <data-dir>

      The restore won't write to a directory that already exists, so first rename or move what remains of data-dir.

    • The etcd guide uses etcdutl snapshot restore, which takes the same options. KX Sensors installs only etcdctl alongside the etcd binary, so the example uses etcdctl snapshot restore. It's deprecated in etcd 3.5, but still supported.

  6. Restore the TP+ logs, depending on what survived:

    • emsLogDir is intact: leave it in place. Processes replay from the restored positions.
    • emsLogDir was lost and you use TP+ log backups: if the TP+ log backup is intact, restore each stream's logs from it, as described in Restore the primary logs from the backup. If the TP+ log backup was lost too, restore emsLogDir from your off-host copy, if you took one. Data received after that copy is lost. At the next start, TP+ finds the empty backup out of sync and disables it, so resynchronize the backup once the system is running again.
    • emsLogDir was lost and you copy the logs yourself: restore each stream's directory from your most recent copy. Data received after that copy is lost.

  7. Start the system by using kxsctl start system.

    If you restored the etcd registry from a snapshot that was taken while the system was running, the registry still records the processes as running. kxsctl start system then prints a Detected stopped process ... with active registry state warning for each process and asks you to confirm. Check that no KX Sensors processes are running on the node, then enter y to continue. If you run the command non-interactively, it stops at this prompt without starting any processes. 8. Check the TP+ logs for Backup position is out of sync with stream or Backup is disabled messages, and confirm that subscribers complete log replay.

Next steps