Skip to content

Set up your EMS streams in ems.yaml

This page describes how to define internal and external EMS streams in ems.yaml.

In this configuration object, you set up your internal and external EMS streams. For each stream, you define the nodes, log level, and the archiving and raft properties. Each stream constitutes a separate RT cluster.

Stream type Description
External stream One external stream is required for each external KXS feed.
Internal stream Only one internal stream (emsint) is allowed in any given KXS installation.

Refer to the meta file in kxs-core/meta/ems.yaml to find out the required data type for each parameter.

The following example shows ems.yaml with one internal stream (emsint) and two external streams: one for feed1 and a second for feed2.

ems:
  values:
    emsint:
      nodes: [ A, B, C ]
      network: kx
      logLevel: DEBUG
      archiving:
        size: 20Gi
        retentionDuration: 18000
        diskUsage: 90
      external: false

    feed1:
      nodes: [ A, B, C ]
      network: kx
      logLevel: DEBUG
      archiving:
        size: 20Gi
        retentionDuration: 18000
        diskUsage: 80
      external: true

    feed2:
      nodes: [ A, B, C ]
      network: kx
      logLevel: DEBUG
      archiving:
        size: 20Gi
        retentionDuration: 18000
        diskUsage: 90
      external: true

The following table describes each attribute you can set for a stream in ems.yaml.

Attribute Description Example
nodes List of nodes that the RT sequencer is running on (must be 1 or 3). [ A, B, C ]
network Deprecated in 3.3. In earlier v3 releases, the Docker overlay network for RT sequencers to attach to. Docker is no longer required to run RT, so this field is no longer used; leave it blank. It's retained only to avoid problems during dynamic upgrade, so you don't need to remove it from an existing ems.yaml. kx
logLevel Logging level for RT sequencer. DEBUG
external Flag indicating whether the stream has external publishers or subscribers. false
tls SSL encryption mode denoting the following: 0 - SSL disabled (default); 1 - Mixed mode, SSL encryption on receipt of external traffic, no encryption on internal traffic; 2 - SSL must be used by all publishers. 0
bridge Flag indicating whether to start the RT bridge q process to receive qIPC traffic. In this configuration, external publishers will not store RT data locally on disk. false
archiving
size Maximum size of all log files in RT. Rolled merged log files that push the total size beyond this limit are garbage collected, oldest first. The supported suffixes are [KMGTP]i (KiB, MiB, GiB, TiB, PiB respectively); for example, use 5Gi for 5GiB. 20Gi
retentionDuration Retention period for merged RT log files in minutes. Rolled merged log files which contain messages older than this (based on the message timestamp) are garbage collected. Set to 0 to disable time-based deletion. 18000
diskUsage Maximum percentage of the available disk space that will be used by RT. When this percentage is exceeded, rolled merged log files are garbage collected starting with oldest first. 80
raft Raft configuration used in multi-node deployments.
heartbeat Raft heartbeat in milliseconds between RT sequencers in a Raft cluster. 10000
chunkSize Size of the Raft or command log in GiB before it is rolled or chunked. 1
logSize Size of the total Raft log to keep in GiB. 5
tp TP+ configuration, used in single node deployments.
backup Flag indicating whether TP log backups are enabled on the stream. The backups are written to .cfgp.tpLogBackupDir. true
checkpointFreq Frequency at which TP state checkpoints are written to, in milliseconds. 10000
flushFreq Frequency at which TP flushes writes to its log, in milliseconds. 1000
bufferSize Size (count) of TP message buffer. 1000
bufferPurgeRatio Ratio of messages to discard when the TP message buffer becomes full. 0.2

The pub and sub fields in process.yaml define the EMS topics that a service class will publish and subscribe to on the emsint stream. For example:

process:
  values:
   - svcClass: RDB
     pub: MdUpdate
     sub: [ MdUpdate, QueryLog, MD, SD ]

Next steps