Skip to content

RT deduplication publisher clients observability

This API returns information about the publishers connected to the RT and pushing data into a deduplication stream.

To call the REST API, you can use:

Endpoint

The $ENDPOINT variable below is defined as the port-forwarded connection to port 6000 of any sequencer nodes.

bash-4.4$ curl http://$ENDPOINT/dedup-rt-clients

Note

This query can be called directly on a sequencer node.

bash-4.4$ curl http://0:6000/dedup-rt-clients

Endpoint

The $ENDPOINT variable below is defined as port 5001 or 5002 at the hostname of any sequencer nodes.

bash-4.4$ cd /opt/kx/app/rt/replicator/clib/
bash-4,4$ ./rest_proxy --endpoint $ENDPOINT --target dedup-rt-clients

Response

The response includes a response header and a payload.

Each deduplication stream is keyed by its dedup-id and watermark. All the publishers pushing data into a given stream are listed in the publishers section with the metrics of the publisher state regarding the deduplication stream.

The payload members are:

group member type description
dedup_id string The name of a deduplication stream.
watermark unsigned The watermark value of the a deduplication stream.
publishers key string The publisher's client name.
last_message datetime The ts member from the message header of the lastly pushed message.
msgs_merged unsigned Number of messages are merged by this publisher into the subscriber stream.
seq_num unsigned The id member from the message header of the lastly pushed message.

Note

The details of the RT message header describes the functonality of all the members can be found in the header.

Example

Endpoint

An example of the $ENDPOINT variable is 0:6000.

bash-4.4$ curl http://$ENDPOINT/dedup-rt-clients | jq .
{
  "result": [
    {
      "dedup_id": "banana",
      "watermark": 103,
      "publishers": [
        {
          "pub-3.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.510844057",
            "msgs_merged": 104,
            "seq_num": 103
          },
          "pub-4.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.359598117",
            "msgs_merged": 0,
            "seq_num": 87
          }
        }
      ]
    },
    {
      "dedup_id": "vanilla",
      "watermark": 103,
      "publishers": [
        {
          "pub-1.kxi-mystream": {
            "last_message": "2025-01-17T09:58:46.286049652",
            "msgs_merged": 57,
            "seq_num": 71
          },
          "pub-2.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.924897920",
            "msgs_merged": 47,
            "seq_num": 103
          }
        }
      ]
    }
  ]
}

Note

First, you must exec onto a sequencer node.

bash-4.4$ curl http://0:6000/dedup-rt-clients | jq .
{
  "result": [
    {
      "dedup_id": "banana",
      "watermark": 103,
      "publishers": [
        {
          "pub-3.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.510844057",
            "msgs_merged": 104,
            "seq_num": 103
          },
          "pub-4.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.359598117",
            "msgs_merged": 0,
            "seq_num": 87
          }
        }
      ]
    },
    {
      "dedup_id": "vanilla",
      "watermark": 103,
      "publishers": [
        {
          "pub-1.kxi-mystream": {
            "last_message": "2025-01-17T09:58:46.286049652",
            "msgs_merged": 57,
            "seq_num": 71
          },
          "pub-2.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.924897920",
            "msgs_merged": 47,
            "seq_num": 103
          }
        }
      ]
    }
  ]
}

Endpoint

An example of the $ENDPOINT variable is kxi-mystream-0:5002.

bash-4.4$ cd /opt/kx/app/rt/replicator/clib/
bash-4,4$ ./rest_proxy --endpoint $ENDPOINT --target dedup-rt-clients | jq .
{
  "result": [
    {
      "dedup_id": "banana",
      "watermark": 103,
      "publishers": [
        {
          "pub-3.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.510844057",
            "msgs_merged": 104,
            "seq_num": 103
          },
          "pub-4.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.359598117",
            "msgs_merged": 0,
            "seq_num": 87
          }
        }
      ]
    },
    {
      "dedup_id": "vanilla",
      "watermark": 103,
      "publishers": [
        {
          "pub-1.kxi-mystream": {
            "last_message": "2025-01-17T09:58:46.286049652",
            "msgs_merged": 57,
            "seq_num": 71
          },
          "pub-2.kxi-mystream": {
            "last_message": "2025-01-17T09:58:47.924897920",
            "msgs_merged": 47,
            "seq_num": 103
          }
        }
      ]
    }
  ]
}