Skip to content

RT log files

This page covers disk performance requirements, archiving parameters, and capacity planning for RT log files, and how RT protects against data loss.

Disk performance requirements

SSD storage is strongly recommended for RT log files. HDDs or network-attached storage (NAS, NFS, iSCSI) introduce significant latency and aren't suitable for RT components handling active sequencing.

RT logs and sequencing stores (Raft logs) should be located on disks separate from database volumes. If collocated on the same physical disk, heavy I/O from the database — especially during end-of-interval write-downs or end-of-day archival — contends with RT's sequencing writes and reads, causing measurable degradation in sequencing latency.

Disk locations

RT writes all classes of data to disk under a single directory, set by emsLogDir in systemParams.yaml and substituted in place of ${EMSLOGDIR}:

  • emsLogDir (default ${KXS_ROOT}/ems/${NODE}): Primary directory for RT data. Contains sequencer logs, Raft state, session files, and all message persistence, organized per stream. This is the most critical directory to provision on fast SSD-backed storage with sufficient space.

For production deployments, it's often worth isolating this directory on a different drive from the kdb+ database to reduce I/O contention.

Changed in 3.3

The separate rtLogDir, rtSeqDir, and rtSesDir parameters are removed; everything now lives under emsLogDir. Remove these entries from your systemParams.yaml when you upgrade.

Application logs

RT application logs are written to the KX Sensors application log directory alongside every other KX Sensors process, in the log of the RT parent process for each stream — for example, kxsRT_A-emsint.log. Replicator logs for publishers and subscribers are written under $KXS_LOG_DIR/kxi_c_sdk_logs, as rt_helper.log files inside directories prefixed rt_helper.. Grep that directory for a process name to match an RT subscriber or publisher to its replicator log.

Changed in 3.3

In earlier v3 releases, RT application logs went to the Docker container logs, rotated by the rtoDockerLogMaxSize, rtoDockerLogMaxFile, and rtoDockerLogCompress system parameters. Those three parameters are removed, and RT logs now follow normal KX Sensors application-log behavior.

Archiving parameters

Archival behavior is configured in ems.yaml under the archiving section. These parameters govern when RT removes old merged log files:

  • size: Defines the maximum size of merged log files for a single RT stream. Supports the suffixes Ki, Mi, Gi, Ti, and Pi. Once exceeded, the oldest merged files are garbage collected. This is the primary mechanism for controlling storage growth per stream.
  • retentionDuration: Time-based retention policy, in minutes. Merged log files whose first message is older (based on message timestamp) than the retention duration are garbage collected. Set to 0N to disable time-based deletion.
  • diskUsage: Maximum fraction of disk that RT is allowed to consume (default: 90%). If exceeded, the oldest merged logs are deleted.

Warning

diskUsage is a last-resort safeguard. Avoid relying on it, since it can cause unexpected data loss.

Capacity planning with archiving

When sizing disk, plan proactively using size and retentionDuration. Don't rely on diskUsage.

For example, given 500 GB of total available disk and two streams with similar load, a safe allocation is 200 GB per stream — a total of 400 GB, or 80% of capacity — leaving roughly 50 GB (10%) of headroom to avoid triggering diskUsage deletion.

General rules:

  • Always choose the size parameter so aggregate usage across all streams stays well below the diskUsage percentage.
  • Align retentionDuration with your business or compliance requirements, but don't let it extend retention beyond what size allows.
  • Monitor disk growth. Use metrics and alerts to ensure usage doesn't exceed 70-80%.

Data loss considerations

Normal cleanup happens through size and retentionDuration, which should be tuned carefully to enforce predictable retention. Emergency cleanup happens when diskUsage is exceeded: RT deletes the oldest logs until usage falls below the threshold. This occurs at the same time on an unpredictable subset of EMS streams in the system, based on q timers, and can cause:

  • Unexpected data loss, since subscribers may lose dropped messages.
  • Incomplete replay history for subscribers, which may cause a process to fail to replay its log on restart.

To protect against this, KX Sensors prefers bringing down all EMS streams on the affected node, based on checks performed by the local RTO process. This check is driven by the following system parameters:

Attribute Description Dynamic Default
rtoChkDiskFreq Retry frequency used for checking disk usage on the RT sequencer log directory, in seconds. N 10
rtoDiskBufPct Safety buffer of disk usage below the archiving limit. Beyond this buffer, RT shuts itself down. N 5

By default, RTO checks disk usage on the mount containing emsLogDir every 10 seconds. It shuts down all RT instances on the node when disk usage exceeds 85% (diskUsage minus rtoDiskBufPct). Set rtoChkDiskFreq to null to disable the check, which KX recommends only where RT logs are isolated on their own disk.

Next steps