RT deduplication publisher clients observability

This API returns information about the publishers connected to the RT and pushing data into a deduplication stream.

To call the REST API, you can use:

port-forwardinglocalhostrest-proxy

Endpoint

The $ENDPOINT variable below is defined as the port-forwarded connection to port 6000 of any sequencer nodes.

bash-4.4$ curl http://$ENDPOINT/dedup-rt-clients

Note

This query can be called directly on a sequencer node.

bash-4.4$ curl http://0:6000/dedup-rt-clients

Endpoint

The $ENDPOINT variable below is defined as port 5001 or 5002 at the hostname of any sequencer nodes.

bash-4.4$ cd /opt/kx/app/rt/replicator/clib/
bash-4,4$ ./rest_proxy --endpoint $ENDPOINT --target dedup-rt-clients

Response

The response includes a response header and a payload.

Each deduplication stream is keyed by its dedup-id and watermark. All the publishers pushing data into a given stream are listed in the publishers section with the metrics of the publisher state regarding the deduplication stream.

The payload members are:

group	member	type	description
dedup_id		string	The name of a deduplication stream.
watermark		unsigned	The watermark value of the a deduplication stream.
publishers	key	string	The publisher's client name.
	last_message	datetime	The ts member from the message header of the lastly pushed message.
	msgs_merged	unsigned	Number of messages are merged by this publisher into the subscriber stream.
	seq_num	unsigned	The id member from the message header of the lastly pushed message.

Note

The details of the RT message header describes the functonality of all the members can be found in the header.

Example

port-forwardinglocalhostrest-proxy

Endpoint

An example of the $ENDPOINT variable is 0:6000.

bash-4.4$ curl http://$ENDPOINT/dedup-rt-clients | jq .
{
  "result": [
    {
      "dedup_id": "banana",
      "watermark": 103,
      "publishers": [
        {
          "pub-3.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.510844057",
            "msgs_merged": 104,
            "seq_num": 103
          },
          "pub-4.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.359598117",
            "msgs_merged": 0,
            "seq_num": 87
          }
        }
      ]
    },
    {
      "dedup_id": "vanilla",
      "watermark": 103,
      "publishers": [
        {
          "pub-1.kxi-mystream": {
            "last_message": "2025-01-17T09:58:46.286049652",
            "msgs_merged": 57,
            "seq_num": 71
          },
          "pub-2.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.924897920",
            "msgs_merged": 47,
            "seq_num": 103
          }
        }
      ]
    }
  ]
}

Note

First, you must exec onto a sequencer node.

bash-4.4$ curl http://0:6000/dedup-rt-clients | jq .
{
  "result": [
    {
      "dedup_id": "banana",
      "watermark": 103,
      "publishers": [
        {
          "pub-3.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.510844057",
            "msgs_merged": 104,
            "seq_num": 103
          },
          "pub-4.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.359598117",
            "msgs_merged": 0,
            "seq_num": 87
          }
        }
      ]
    },
    {
      "dedup_id": "vanilla",
      "watermark": 103,
      "publishers": [
        {
          "pub-1.kxi-mystream": {
            "last_message": "2025-01-17T09:58:46.286049652",
            "msgs_merged": 57,
            "seq_num": 71
          },
          "pub-2.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.924897920",
            "msgs_merged": 47,
            "seq_num": 103
          }
        }
      ]
    }
  ]
}

Endpoint

An example of the $ENDPOINT variable is kxi-mystream-0:5002.

bash-4.4$ cd /opt/kx/app/rt/replicator/clib/
bash-4,4$ ./rest_proxy --endpoint $ENDPOINT --target dedup-rt-clients | jq .
{
  "result": [
    {
      "dedup_id": "banana",
      "watermark": 103,
      "publishers": [
        {
          "pub-3.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.510844057",
            "msgs_merged": 104,
            "seq_num": 103
          },
          "pub-4.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.359598117",
            "msgs_merged": 0,
            "seq_num": 87
          }
        }
      ]
    },
    {
      "dedup_id": "vanilla",
      "watermark": 103,
      "publishers": [
        {
          "pub-1.kxi-mystream": {
            "last_message": "2025-01-17T09:58:46.286049652",
            "msgs_merged": 57,
            "seq_num": 71
          },
          "pub-2.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.924897920",
            "msgs_merged": 47,
            "seq_num": 103
          }
        }
      ]
    }
  ]
}