TP+ debugging and observability¶

This page explains the TP+ logging facilities, the diagnostic helper functions available for inspecting logs and monitoring publisher and subscriber activity, and how to configure, monitor, and restore from TP+ log backups.

Log categories: Use the TP, TPP, TPPUB, and TPSUB logging facilities to filter startup/registration, core buffer, publisher, and subscriber events respectively.
Diagnostic helpers: The EMS message layer exposes d.getMsg, d.getLogInfo, d.getMsgPos, and d.readLog functions so administrators can peek into log files, resolve positions to file offsets, and export message tables for manual inspection.
TP+ helpers: The TP+ core exposes scanLog, scanLogRcv, scanLogEOL, showLog, stats, and parsePos so you can replay and inspect parts of the log, verify stream positions, and sanity-check recovery data before and after operational changes.

TP+ log backups¶

On a single-node system, there's no other node to fail over to and no other copy of the TP+ logs. The TP+ logs support log replay for processes as part of system recovery. If the disk that holds them fails, you lose the data received since the last EOI.

KX recommends that you enable TP+ log backups for every stream on single-node systems, as the primary protection for the TP+ logs. For how TP+ log backups fit into a complete backup procedure, see Back up and restore a single-node system.

When TP+ log backups are enabled, TP+ writes every message to a second set of logs in tpLogBackupDir, in the same format as the primary logs, and flushes both sets on the same cycle (flushFreq, by default every second). If the primary logs are lost, you can restore them from the backup and replay from it, losing at most about one flush interval of data.

Note

Archiving parameters that are set for the stream apply to both the primary and backup TP+ logs. The backup isn't a long-term archive: set retentionDuration and size so that the logs cover at least the time since your last backup of the IDB.

Configuration¶

.cfgp.tpLogBackupDir: system parameter representing the directory containing the TP+ backups for all streams. Each stream's backup is written to <tpLogBackupDir>/<stream>, with the same layout as <emsLogDir>/<stream>. The default is ${KXS_ROOT}/tpp_bkup. Changing it requires a restart.
.cfgp.tpExitOnBackupMismatch: system parameter indicating whether TP+ exits if the backup is out of sync with the primary log at startup. The default is false. See How TP+ checks the backup.
backup: boolean parameter inside the tp configuration field of ems.yaml indicating whether TP+ backups should be enabled for the stream.

Warning

Set tpLogBackupDir to a directory on a different physical disk from emsLogDir. The default location is under $KXS_ROOT, which is usually the same disk as the primary logs, so a single disk failure loses both copies.

How TP+ checks the backup¶

TP+ checks the backup only when it opens the stream at startup. It doesn't check again while it's running. At startup:

If the backup's last position doesn't match the primary log and tpExitOnBackupMismatch is false, TP+ logs Backup position is out of sync with stream and Backup is disabled at error level, and continues without a backup.
If the backup's last position doesn't match and tpExitOnBackupMismatch is true, TP+ logs a fatal error and exits without an automatic systemd restart. The stream is unavailable until you resolve the mismatch.
If TP+ can't open the backup at all, for example because the directory is missing or the disk isn't mounted, and tpExitOnBackupMismatch is false, TP+ continues without a backup and doesn't log a backup-specific message. With true, TP+ fails to start.

After TP+ disables the backup, the backup stays disabled until TP+ restarts. At the next restart, the backup is usually further behind and is disabled again, so it stays disabled until you resynchronize it.

Common causes of a mismatch are:

Enabling backup on a stream that already has logs, without first seeding the backup directory.
Replacing or clearing the disk that holds tpLogBackupDir.
Restoring either emsLogDir or tpLogBackupDir without the other.

Setting tpExitOnBackupMismatch to true favors a guaranteed backup over availability; false favors availability. If you leave it at false, monitor the backup as described in the next section.

Monitor TP+ log backups¶

Alert on the Backup position is out of sync with stream and Backup is disabled log messages in the TP+ logs.
Because an open failure isn't logged, also periodically confirm that the backup is active. In a q session connected to the TP+ process (kxsTP_<node>-<stream>), a null .tpp.bkh means that the backup is disabled. While the backup is active, comparing .tpp.state[] with .tpp.bkstate[] shows the last position written to the primary and backup logs. Check .tpp.bkh first, because .tpp.bkstate[] signals an error when the backup is disabled.
Monitor free space on the volume that holds tpLogBackupDir.

Resynchronize the backup¶

Use this procedure when the backup has been disabled, or before you enable backup on a stream that already has logs.

Warning

Resynchronizing copies the primary logs over the backup. If the primary logs are behind the backup, for example because the primary disk was replaced, the backup is the only complete copy. In that case, restore the primary logs from the backup instead.

Stop the system by using kxsctl stop system.
Delete the stream's backup directory, and replace it with a copy of its primary log directory:
```
rm -rf <tpLogBackupDir>/<stream>
rsync -a <emsLogDir>/<stream>/ <tpLogBackupDir>/<stream>/
```
Copy into an empty directory

Delete the backup directory before you copy. If you copy over it, rsync can skip files that have changed, which leaves the backup out of step with the primary logs without reporting an error.
Start the system by using kxsctl start system.
Check that the backup is in step with the primary logs:
- The TP+ logs don't contain a Backup position is out of sync with stream message for the stream.
- Both directories hold the same session number. The following command prints nothing if they match:
```
cmp <emsLogDir>/<stream>/.session <tpLogBackupDir>/<stream>/.session
```
- Both directories hold the same files. The following command prints nothing if they match:
```
diff <(ls <emsLogDir>/<stream>) <(ls <tpLogBackupDir>/<stream>)
```

Restore the primary logs from the backup¶

Use this procedure when the primary logs in emsLogDir are lost or damaged and the backup is intact.

Stop the system by using kxsctl stop system.
For each stream, delete what remains of the primary log directory, and replace it with a copy of the backup directory, preserving file names:
```
rm -rf <emsLogDir>/<stream>
rsync -a <tpLogBackupDir>/<stream>/ <emsLogDir>/<stream>/
```
As when you resynchronize the backup, copy into an empty directory rather than over existing files.
Start the system by using kxsctl start system. The primary and backup logs now match, so TP+ reopens the backup, and subscribers replay from their restored positions.