Skip to content

RT stream log archival configuration

RT stream log archival rules can be set using global config values. This allows the user to configure the retention duration, max log size and max disk usage percentage for RT stream log files according to their required policies.

global:
  rtConfig:
    retentionDuration: 10080
    maxLogSize: 5Gi
    maxDiskUsagePercent: 90

retentionDuration

Variable Type Example Default
retentionDuration int 7200 10080

Retention period for merged RT log files in minutes. Rolled merged log files which contain messages older than this (based on the message timestamp) are garbage collected.

Set to 0 to disable time-based archival.

maxLogSize

variable type example default
maxLogSize string 10Gi

Maximum size of all log files in RT. Rolled merged log files which push the total size beyond this limit are garbage collected, oldest first.

The supported suffixes are [KMGTP]i (KB, MB, GB, TB, PB respectively), for example use 5Gi for 5GB.

Warning

The size of the disk associated with each RT pod and each subscriber MUST be larger than the maxLogSize defined.

maxDiskUsagePercent

variable type example default
maxDiskUsagePercent int 80 90

Maximum percentage of the available disk space that will be used by RT. When this percentage is exceeded rolled merged log files are garbage collected, oldest first.

RT chart

Stream log archival settings can be configured for the RT chart specifically. This is done by adding a similar block to below in your values file.

rt:
  rt-archiver:
    rtConfig:
      retentionDuration: 10080
      maxLogSize: 5Gi
      maxDiskUsagePercent: 90

Sizing PVCs for RT stream log files

RT stream log files are stored in a number of PVCs in Insights:

  • 3 x rt-north
  • 3 x rt-south
  • spwork (one per pipeline)
  • 3 x dap-rdb
  • 3 x dap-idb
  • 3 x dap-hdb
  • sm

Example PVC Sizes

As a rough rule of thumb to size the PVCs (assuming SP is performing a simple passthrough):

  • rt-north = rt-south
  • spwork = rt-north + rt-south
  • everything else = rt-south

RAFT configuration

In the KX Insights Platform, the 3-node RT Stream implementation uses a RAFT consensus engine which maintains two logs:

  • A RAFT log - contains meta data about the merged stream i.e. all of the header information to allow the node to participate in the RAFT cluster, join and rejoin an already running cluster, call a leader election and to vote in a leader election.

  • A command log - contains the instructions that where submitted to the raft log. The command log can be thought of very much like a tickerplant log and it is this log which is consumed by subscribers.

Both the RAFT and command logs can be rolled once they reach a certain size. This allows the cluster to operate continuously, as rolled logs can be deleted or archived as they age and become obsolete over time.

RAFT uses the following environment variables in the sequencer configuration:

    seq:
      rt:
        env:
        - name: RAFT_CHUNK_SIZE
          value: "1"
        - name: RAFT_LOG_SIZE
          value: "10"
        - name: RAFT_INTERNAL_TIMER
          value: "10"

RAFT_CHUNK_SIZE

The size of RAFT or command log in GiB before it is rolled (or chunked).

variable type example default
RAFT_CHUNK_SIZE float 1 1

RAFT_LOG_SIZE

The size of the total RAFT log to keep in GiB. Only completed RAFT logs are discarded, so in this case (RAFT_LOG_SIZE=10 and RAFT_CHUNK_SIZE=1), it is possible that you could use up to 11GiB before it is trimmed back to 10GiB.

RAFT log deletion is taken care of by RAFT itself using the above environment variables. Command logs are deleted by the RT archival policy and the subscriber options.

variable type example default
RAFT_LOG_SIZE int 10 10

RAFT_INTERNAL_TIMER

This is the time in milliseconds for an internal batch timer. This publishes a batch of messages every "batch internal time", regardless of how quickly messages are being submitted to the RAFT log. It works the same as tick.q when it is in batched mode. Due to a constraint of RAFT, this must be reasonably larger than the expected round trip between nodes.

variable type example default
RAFT_INTERNAL_TIMER int 10 10