Skip to content

Disaster recovery

This page explains how to recover service when two nodes of a three-node KX Sensors cluster are lost, by switching the cluster to run on the surviving node alone, and how to return the cluster to three nodes once the lost nodes are back online.

Supported cluster configurations

The disaster recovery scripts support only two transitions: from three nodes to one node, and from one node back to three nodes. Other cluster sizes and transitions, for example a five-node cluster, or recovering to two nodes, aren't supported yet.

Overview

A KX Sensors cluster relies on etcd for service discovery and on RT for reliable transport. Both use the Raft algorithm to reach consensus, and Raft needs a majority of members to make progress. In a three-node cluster, the majority is two, so if you lose two of the three nodes, the surviving node can't elect a leader, and etcd and RT stop making progress.

To recover service, you collapse the cluster down to the surviving node, which can then operate on its own as a majority of one. Once the lost nodes are back online, you grow the cluster back to three nodes.

flowchart LR
    A["3 nodes"] -- "2 nodes lost: 3to1.sh" --> B["1 node"]
    B -- "lost nodes back online: 1to3.sh" --> C["3 nodes"]

This procedure causes data loss

The procedure prioritizes service availability over data retention. Both phases perform a hard reset of RT, which discards RT data that hasn't been written to the database. Everything received since the last EOI is lost.

Before you begin

  • Stop all ingestion clients before you run either script. Both scripts prompt you to confirm this before they do anything.
  • Run both scripts on the surviving node, as the KX Sensors install user, with the KX Sensors environment sourced (KXS_ROOT, KXS_DISC_DATA_DIR, and KXS_BASE_PORT). The scripts use the user or system systemd instance, depending on the user that runs them.
  • Before you run 3to1.sh, make sure that the two lost nodes are down or unreachable.
  • Before you run 1to3.sh, make sure that the recovered nodes are back online and reachable from the surviving node over passwordless SSH. 1to3.sh runs every action on a recovered node through ssh.
  • Capture each script's output to a file, for example ./3to1.sh -n A -a 2>&1 | tee dr-3to1.log. The scripts log every step, with a timestamp, to standard output only, and you need this output to diagnose a failure.

Deployments that use Docker (before 3.3)

On deployments that still run RT in Docker containers, run the scripts only on a host that runs a single KX Sensors environment. The scripts select the RT containers to remove by their container label, which isn't scoped to one environment, so on a host with several environments they also remove the other environments' RT containers.

The disaster recovery scripts

The scripts are installed in ${KXS_ROOT}/packages/kxs-core/src/scripts/Linux/dr/.

Script Purpose
3to1.sh Switches the cluster from three nodes to the surviving node.
1to3.sh Returns the cluster from one node to three nodes.
hardReset.sh Discards RT state and restarts. Both phases call it: -n <node> resets one node, and no -n resets all nodes. Before it deletes each directory, it prompts you to back it up.
common.sh Shared helpers and etcd configuration that the other scripts load. Don't run it directly.

Switch from three nodes to one

Run 3to1.sh on the surviving node after two nodes are lost, while those nodes are down or unreachable:

./3to1.sh -n <surviving node> -a
Syntax 3to1.sh -n <node> [-a] [-b] [-c] [-d] [-e] [-f] [-g]
Flags -n <node>
The surviving node.
-a
Runs all steps. Combined with step flags, runs all steps except those listed.
-b to -g
Runs only the listed steps.

The script runs the following steps, in order. Each step depends on the steps before it.

Step Action
-b Saves a snapshot of the etcd database to ${KXS_DISC_DATA_DIR}/dr-snapshot.db.
-c Restores the snapshot as a new single-member etcd cluster in a staging directory, without touching the live data directory, and creates a new cluster token.
-d Stops etcd, moves the restored data directory into place, updates etcd-conf.yml for a one-member cluster with the new cluster token, points KXS_DISC_ENDPOINTS at the surviving node, reloads systemd, and starts etcd and qetcd.
-e Stops KX Sensors on the surviving node, and removes any remaining RT Docker containers. On deployments without Docker, this removal does nothing.
-f Starts TS, and marks every node except the surviving node as DELETED in node state.
-g Performs a hard reset of RT on the surviving node by running hardReset.sh -n <node>, then restarts.

When the script completes, the system runs on the surviving node, with EMS streams switched from RT to TP+. The cluster has no redundancy in this state.

Why the switch discards RT data

When two of three nodes are lost, there's no reliable way to know how far the surviving node had progressed relative to the lost nodes. It may be missing log entries that the other nodes had already accepted. If it kept its RT logs, processes that tried to replay from a position it never received would fail, and it would be hard to tell which data is present in the system.

The hard reset in step -g removes that uncertainty. It gives a known starting point: everything since the last EOI is lost, and nothing else.

Why the switch creates a new etcd cluster

Step -c creates a unique cluster token, kxs-etcd-dr-<timestamp>, saves it to ${KXS_DISC_DATA_DIR}/dr-cluster-token, and step -d writes it to etcd-conf.yml. As a result:

  • The surviving node runs a new etcd cluster with a new cluster ID. If you power a lost node back on, it can't rejoin this cluster and bring back out-of-date state.
  • 1to3.sh can't restart the lost nodes into the cluster as they are. Instead, it clears each recovered node's etcd data directory and gives the node the surviving node's cluster token, so each one joins empty and copies its state from the surviving node.

The script creates a new token every time you run it, so repeated recoveries never reuse a token.

Resume after a failure

If 3to1.sh fails partway, rerun the remaining steps with their flags. The order of flags doesn't matter.

./3to1.sh -n A -a        # Run all steps
./3to1.sh -n A -a -b     # Run all steps except -b, for example because the snapshot was already taken
./3to1.sh -n A -g        # Run only step -g, the hard reset

Run individual steps only when you know their dependencies are already complete. For example, -c needs the snapshot from -b, and -d needs the restored directory from -c.

The script has no help flag. If you run it with no arguments or with an unrecognized flag, it prints its usage and exits with a non-zero status.

Return from one node to three

Run 1to3.sh on the surviving node once the two lost nodes are back online. You must have run 3to1.sh first.

./1to3.sh -n <surviving node> [-t <timeout>]
Syntax 1to3.sh -n <node> [-t <timeout>]
Flags -n <node>
The surviving node.
-t <timeout>
Time in seconds to wait for the system to stop after the EOI. The default is 1200.

The script runs the following steps, in order:

  1. Checks that the surviving node's etcd cluster has exactly one member, and stops if it doesn't. More than one member means that the lost nodes were powered on and rejoined, or that 1to3.sh has already run.
  2. Triggers an EOI to write all in-memory data to disk. Any data ingested after this point is lost.
  3. Stops KX Sensors processes on the recovered nodes, then stops the surviving node, waiting up to the -t timeout.

  4. For each recovered node:

    1. Stops etcd and qetcd.
    2. Adds the node to the etcd cluster.
    3. Clears the node's etcd data directory, because a joining member must start empty.
    4. Updates the node's etcd-conf.yml with its member name, data directory, the surviving node's cluster token, and the cluster membership.
    5. Reloads systemd, starts etcd, and confirms the node is a member.
    6. Adds the node to KXS_DISC_ENDPOINTS in the surviving node's kxsenv.
    7. Starts qetcd.

  5. Starts TS on the surviving node, and sets the recovered nodes to ENABLED in node state, so that the hard reset and restart in the next step include them.

  6. Performs a hard reset of RT on all nodes by running hardReset.sh without -n, then restarts.
  7. Sets the recovered nodes to DISABLED in node state, so that they don't serve queries or become failover candidates until their databases are synchronized with the surviving node.

Resume after a failure

1to3.sh doesn't support resuming, and rerunning it after a partial run isn't safe as it is. The check in step 1 stops the script if any node was already added, and step 4 appends to KXS_DISC_ENDPOINTS, so a partial run can leave duplicate endpoints. Before you rerun it, review and correct the etcd membership, the node state, and KXS_DISC_ENDPOINTS in kxsenv.

After recovery

When 1to3.sh completes, the cluster has three nodes, but the recovered nodes are still DISABLED. To finish the recovery:

  1. On the surviving node, check that etcd has three started members:

    <etcd directory>/etcdctl --endpoints=<scheme>://<surviving node host>:${KXS_BASE_PORT} member list

    The etcd directory is the one that holds the etcd binary, by default ${KXS_ROOT}/bin/etcd. Use https as the scheme if ETCD_SSL_CLIENT is true in kxsenv, and add the --cacert, --cert, and --key flags. Otherwise, use http.

  2. Check the node state:

    kxsctl show nodeState

  3. Synchronize the recovered nodes' databases with the surviving node by using DBC from a maintenance console.

  4. Once the databases are synchronized, enable the recovered nodes so that they serve queries and can act as failover candidates:

    kxsctl enable <recovered nodes>

Things to keep in mind

  • 3to1.sh step -d and 1to3.sh step 4 delete etcd data directories that belong to the previous cluster. This is expected during recovery, but confirm that you're on the intended node before you run them.
  • The scripts edit etcd-conf.yml, which sits alongside the etcd binary, in place, rather than rewriting the systemd unit file. The etcd client port is KXS_BASE_PORT, and the peer port is KXS_BASE_PORT + 1. The scheme and TLS settings follow the ETCD_SSL_* settings in kxsenv.

Next steps