Skip to content

Set up a message queue

This page explains how to configure and deploy Message Queue (MQ) service class instances.

Overview

In KX Sensors, the concept of notifications applies when one wants to push updates to external clients, rather than having clients poll the system by pulling information (via queries). In KX Sensors v2, two types of notifications existed:

  • Asynchronous: in v2, delivery of these notifications was not guaranteed. They were suitable for statistical or monitoring purposes in which the occasional missing notification was not critical. This type of notification is currently not supported in v3.
  • Synchronous: in v2, delivery of these notifications was guaranteed. In v3, this functionality exists, but the terminology of subscribers and notifications is no longer used. Instead, a queue of messages is distributed to a pool of message workers by a message broker. The Message Queue (MQ) service class acts as the message broker to provide this functionality. MQ pushes messages to a list of registered workers. It aims to deliver every message to at least one worker and have it (optionally) acknowledged. Unacknowledged messages are retried indefinitely. The queue of messages is transmitted via RT, which means messages are persisted to disk for durability, and in a multi-node cluster, are redundantly available across nodes.

The following section describes what is required to set up a message queue.

Configure mq.yaml and feed.yaml

Each MQ process on a node is responsible for one queue, so the number of MQ processes needs to scale with the number of queues. In a multi-node context, the ownership of a queue can be split across multiple nodes. The MQ service class should be included in the manifest file for any node which is part of the HA cluster. Since MQ is an HA client, there is the concept of a primary MQ and potentially multiple secondaries. The primary MQ is responsible for subscribing to the input stream and forwarding messages to the client. Information on where the primary MQ resides is stored in discovery to allow the client to connect to the correct MQ. Secondary MQs do not subscribe to the input stream.

Message Queue configuration is driven by two configuration files:

  • mq.yaml
  • feed.yaml

mq.yaml contains the definitions of message queues and their properties.

mq:
  values:
    Queue1:
      feed: Queue1

In the example above, the message queue Queue1 is configured to use feed Queue1. Message queues are conceptually like feeds in that they have an input source and an output target of data. The output data for MQ is the acknowledgment stream, which keeps track of messages that have been acknowledged by the client.

A message queue's input source and HA properties are in turn defined by the feed properties in feed.yaml. There should be an entry for each queue in feed.yaml. In this example, that is Queue1. Each queue will have a stream and topic which the relevant MQ process will subscribe to (streamIn and topicIn respectively). Each queue will also have a stream and topic on which the relevant MQ process will publish acks that have been received (streamsOut and topicsOut respectively). These streams will need to exist in ems.yaml (see the relevant section for more details).

feed:
  values:
    Queue1:
      id: 110
      sc: mq
      nodes: [ A, B ]
      procs: [ kxsMQ_A1, kxsMQ_B1 ]
      streamIn: emsint
      topicIn: Queue1
      streamsOut: mqack
      topicsOut: Queue1Ack
      desc: Test queue

streamIn and streamsOut must not be the same within a queue. If there are multiple queues, the same streamsOut can be used for each provided topicsOut is distinct per queue (the same goes for streamIn and topicIn).

Next steps