Skip to content

Perform a hard reset of RT in KX Sensors

This page explains how to perform a hard reset of RT, which clears all RT log and state data across every node in the cluster.

This procedure causes data loss

A hard reset deletes RT's logs and Raft state, along with the stream-processor checkpoints and watermarks that depend on them. Messages that subscribers haven't yet consumed are lost, and replay history is discarded. Treat it as a last resort, back up everything the procedure deletes, and where possible contact KX to carry it out with you — KX normally performs this procedure on a customer's behalf.

Before you begin

The procedure operates on four directories. Resolve each one from your own systemParams.yaml rather than assuming the defaults, since these are commonly relocated onto dedicated volumes.

Parameter Description Default
emsLogDir Directory for EMS logs, substituted in place of ${EMSLOGDIR}. Holds RT's sequencer logs, Raft state, and session files. ${KXS_ROOT}/ems/${NODE}
opDataDir Process-specific operational data directory, substituted in place of ${OPDATA}. Holds stream-processor checkpoints and, by default, watermarks. ${KXS_ROOT}/opdata_${NODE}/${INST}
idbDir Root of the on-disk IDB. ${KXS_ROOT}/db/idb
wmDir Watermark directory. Deployments sometimes move this outside opDataDir. ${OPDATA}/wm

Procedure

Perform every step on each node in the cluster.

  1. Stop all nodes.

    kxsctl stop system

  2. Back up the contents of emsLogDir. Everything deleted below lives under the directories listed above, so take copies before you continue.

  3. Delete the full contents of emsLogDir.

  4. Delete the stream-processor checkpoints and watermarks under opDataDir. Each process has its own subdirectory holding a chkp folder and a wm folder; remove both from every process. The watermarks SM uses to subscribe to the emsint stream, eox and seq, are among those removed here.

    root@kxsa:~/kxs/opdata_A# ls -ld */chkp */wm
drwxr-xr-x. 2 root root 71 May  8 14:03 kxsMDL_A/chkp
drwxr-xr-x. 2 root root 30 May  8 14:03 kxsMDL_A/wm
drwxr-xr-x. 2 root root 80 May  8 14:03 kxsSDL_A1/chkp
drwxr-xr-x. 2 root root 30 May  8 14:03 kxsSDL_A1/wm
drwxr-xr-x. 2 root root 80 May  8 14:03 kxsSDL_A2/chkp
drwxr-xr-x. 2 root root 30 May  8 14:03 kxsSDL_A2/wm

root@kxsa:~/kxs/opdata_A# rm -rf */chkp */wm

  5. If wmDir has been changed from its default of ${OPDATA}/wm, delete that directory as well. The previous step only reaches watermarks stored under opDataDir, so a relocated wmDir survives it.

    For example, where wmDir is set to /root/kxs/db/wm:

    root@kxsa:~# rm -rf /root/kxs/db/wm

  6. Reset the position stored in IdbStatus. Run the following from a maintenance console. This sets the position to null, which the system treats as zero.

    {p set update pos:0N from get p:` sv .cfgp.idbDir,`data,x}each`IdbStatus.0`IdbStatus.1

  7. Start the system, or continue with the upgrade that required the reset.

    kxsctl start system

Note

Clearing the watermarks and resetting the IdbStatus position are manual steps. KX intends to provide a supported mechanism for both in a future release.

Next steps