Skip to content

Set up your feeds in feed.yaml

This page explains how to define your data feeds in feed.yaml, and assign them to the appropriate nodes.

In this configuration object, you define your data feeds and assign them to the appropriate nodes. A feed represents a stream of sensor data from an external client to be ingested by one or more KX Sensors nodes. In feed.yaml, you must define the following for each data feed:

  • Feed name
  • Feed ID
  • EMS stream on which input data is received
  • EMS stream on which output data is published
  • EMS topic on which input data is received
  • EMS topic on which output data is published
  • Service class the feed relates to
  • Processes the feed relates to

The following example shows the setup for three feeds distributed over four nodes (A, B, C, D). feed1 and feed2 (IDs 100 and 101 respectively) consist of noncoherent sensor data coming from one or more sources, and each is directed to a pair of nodes. For each feed, either configured node in the pair can act as either primary or secondary. The master data feed, on the other hand, consists of mutation operations performed by the MDL in response to any operations (internal or external) that require adjustments to master data. This feed can be run on any of the four nodes.

feed:
  values:
    MD:
      id: 0
      streamIn: mutreqs
      streamsOut: emsint
      topicIn: MD
      topicsOut: MD
      sc: mdl
      nodes: [ A, B, C, D ]
      procs: [ kxsMDL_A, kxsMDL_B, kxsMDL_C, kxsMDL_D ]
      desc: Master data feed (API calls to MDL)

    feed1:
      id: 100
      isCoherent: false
      streamIn: feed1
      streamsOut: emsint
      topicIn: feed1
      topicsOut: SD1
      sc: sdl
      nodes: [ A, B ]
      procs: [ kxsSDL_A1, kxsSDL_B1 ]

    feed2:
      id: 101
      isCoherent: false
      streamIn: feed2
      streamsOut: emsint
      topicIn: feed2
      topicsOut: SD2
      sc: sdl
      nodes: [ C, D ]
      procs: [ kxsSDL_C1, kxsSDL_D1 ]
Attribute Definition Example
name Feed name. feed1
id Unique numeric identifier for the feed. In HA commands, either this identifier or the feed name can be used to specify a feed. 100
isCoherent Indicates whether the feed is coherent. A coherent feed is a feed that can be safely processed simultaneously by separate nodes because no master data, such as a sensor ID, is being auto-generated for an unknown entity. false
isMediated Indicates whether the feed is actively mediated by a non-coherent secondary node or nodes. Mediation is the process by which the secondary node compares the feed that was replicated from the primary node with the feed that it received directly from the client to ensure that they are identical. This option is not applicable to your master data feed. false
isActiveSec Set to true for all feeds except the master data feed. true
isEpic Indicates if process may engage in long-running activity. true
compFn Comparison function used to compare relative position within the stream of two fingerprints. .ha.comp
streamIn EMS stream containing the input feed topic. feed1
streamsOut EMS stream(s) containing the output topic(s) that an SP publishes to. Must be parallel to, or be able to be scalar extended to, the size of topicsOut. emsint
topicIn Feed topic corresponding to the incoming feed. feed1
topicsOut EMS output topics to which the SP publishes. There can be more than one output topic for a given incoming feed. SD1
dbIgnoreTopics Output topics that DBW should ignore when writing the data from the feed topics (ignored automatically if they are not in emsint). preSD1
streamDefer EMS stream containing deferral topic. defer
streamRelease EMS stream containing release topic. If streamDefer is specified, this must be specified as well. defer
nodes Specifies the nodes supporting this feed, either as primary or secondary, in preferred sequence. [ A, B ]
failoverNodes Specifies the failover nodes in preferred sequence. Usually the same as nodes, but it can be a subset of it. The failover preference order is determined by the order in the nodes column. The failover node list defaults to the nodes value if not specified. [ A, B ]
queryNodes Nodes that support queries of data for the feed. Defaults to nodes if not specified. [ A, B ]
sc Service class responsible for feed ingestion. sdl
procs Process instances responsible for ingestion. May be omitted if only one process of type sc exists on the node. If two or more feeds are attached to the same node, you can define a separate process for each feed depending on the node that it is attached to. The naming convention is SDL_<NODE><NUMBER>. kxsSDL_A1
skewThr Maximum tolerated temporal distance between arrival of message at different nodes, in seconds. 10
phiThr Probability distribution of heartbeat interarrival times. 2.5
phiMeanOfs Acceptable heartbeat fluctuation from mean, in msec. 2200
phiMinDev Minimum standard deviation for normal distribution of heartbeat interarrival times, in msec. 100
desc No functional purpose, only used to document configuration entries. Master data feed (API calls to MDL)

Next steps