Look up the status of feeds and nodes¶
This page describes feed states and modes in KX Sensors and the commands available for looking up feed and node health.
There are several commands that you can use for looking up the state of feeds and nodes in the system.
Look up your feed states and modes¶
Feed states describe the current health of a feed. KX Sensors supports the following states:
| State | Description |
|---|---|
| UNKNOWN | Unknown |
| NOMINAL | Nominal or Normal |
| LAGGING | The feed is not healthy. For example, data is arriving too slowly, there is message loss in the feed, the heartbeats are not being received as expected or the node has failed. |
| FAILOVER | The primary for the feed is not the configured default for primary; i.e., at some point in the past the feed has failed over. |
| FLAGGING | Failover and lagging |
| ABNORMAL | Abnormal |
Feed modes describe the current activity of a feed on a given node.
| Mode | Description |
|---|---|
| UNKNOWN | Unknown |
| INACTIVE | Node does not process feed |
| ACTIVE | Node is processing coherent feed |
| PRIMARY | Node is primary for non-coherent feed |
| SECONDARY | Node is secondary for non-coherent feed |
| PRISEC | Node is transitioning from primary to secondary for non-coherent feed |
| SECPRI | Node is transitioning from secondary to primary for non-coherent feed |
| ACTSYNC | Node processes coherent feed and is in recovery mode |
| PRISYNC | Node is primary for non-coherent feed and is in recovery mode |
| SECSYNC | Node is secondary for non-coherent feed and is in recovery mode |
Look up the state of a feed¶
You can look up the central state of one or more feeds in your environment by means of the sysstate command. It shows one record for each unique node/feed pair. For example, if you have two feeds on two nodes, you will see four records when running sysstate.
For each node/feed combination, sysstate shows:
- the feed ID
- the node that the feed is currently assigned to
- the feed name
- the source node (that is, the node that was responsible for the change)
- the timestamp of the last HA event
- the change counter (a system-generated transaction number for each change of state)
- the current mode of the feed on the node (PRIMARY, SECONDARY, etc.)
- the last HA event (AUTOFOVFEED, HACTL, etc.)
| Syntax | `sysstate;`<feeds> |
|---|---|
| Optional | feeds (either name or ID) |
| Arming required | No |
| Confirmation required | No |
q).maint.ha[`sysstate]
2025.09.03 21:08:18.890 DEBUG [mymaint] DISC Fetching key /config/haState
feed node| feedName sndNode ts cc mode event
---------| -----------------------------------------------------------------
0 A | MD B 2025.08.30D02:13:02.617429167 4 SECONDARY AUTOFOVFEED
0 B | MD B 2025.08.30D02:13:02.617429167 4 PRIMARY AUTOFOVFEED
100 A | feed1 B 2025.09.02D13:36:03.880017270 4 SECONDARY AUTOFOVFEED
100 B | feed1 B 2025.09.02D13:36:03.880017270 4 PRIMARY AUTOFOVFEED
200 A | REPL A 2025.09.03D12:30:55.020216449 4 SECONDARY AUTOFOVFEED
200 B | REPL A 2025.09.03D12:30:55.020216449 4 PRIMARY AUTOFOVFEED
| Line | Description |
|---|---|
| Line 1 | MD (ID 0) on node A is operating as secondary. The event that caused the last state change was an automatic failover triggered by node B (sndNode). |
| Line 2 | MD (ID 0) on node B is operating as primary. The event that caused the last state change was an automatic failover triggered by node B (sndNode). |
| Line 3 | feed1 (ID 100) on node A is operating as secondary. The event that caused the last state change was an automatic failover triggered by node B (sndNode). |
| Line 4 | feed1 (ID 100) on node B is operating as primary. The event that caused the last state change was an automatic failover triggered by node B (sndNode). |
| Line 5 | REPL (ID 200) on node A is operating as secondary. The event that caused the last state change was an automatic failover triggered by node A (sndNode). |
| Line 6 | REPL (ID 200) on node B is operating as primary. The event that caused the last state change was an automatic failover triggered by node A (sndNode). |
An HA event is any action that causes a change in state of a node. HA events display when you run the sysstate command.
| Code | Description |
|---|---|
| HACTL | A default state set by the HA control logic during start-up and initialization. |
| UNUSED1 | Was RS: RS state transition |
| AUTOFOVFEED | Automatic failover due to feed process |
| AUTOFBKFEED | Automatic failback due to feed process |
| AUTOFOVPROC | Automatic failover due to process/hardware failure |
| AUTOFBKPROC | Automatic failback due to process/hardware failure |
| UNUSED2 | Was AUTOFOVQR: Automatic fail-over due to QR failure |
| UNUSED3 | Was AUTOFBKQR: Automatic fail-back due to QR failure |
| MANFOVFEED | Manual failover of feed |
| MANFBKFEED | Manual failback of feed |
| MANFOVPROC | Manual failover of process |
| UNUSED4 | Was NOQUORUM: Primary relinquish due to quorum loss |
Look up the process state of a feed¶
This command shows the same data as the sysstate command as well as the process/service class for the feed.
Note
sysstate is based on state information received from discovery while procstate is based on state information received directly from your data loaders. Ideally, both commands should always show identical information. However, during a mode change when a node is transitioning from primary to secondary or from secondary to primary, your procstate state may not match your sysstate state.
| Syntax | `procstate;`<feeds> |
|---|---|
| Optional | feeds (either name or ID) |
| Arming required | No |
| Confirmation required | No |
q).maint.ha[`procstate]
feed proc | node feedName sndNode ts cc mode event
------------------|--------------------------------------------------------------------------
0 kxsMDL_A | A MD B 2025.08.30D02:13:02.617429167 4 SECONDARY AUTOFOVFEED
0 kxsMDL_B | B MD B 2025.08.30D02:13:02.617429167 4 PRIMARY AUTOFOVFEED
100 kxsSDL_A1 | A feed1 B 2025.09.02D13:36:03.880017270 4 SECONDARY AUTOFOVFEED
100 kxsSDL_B1 | B feed1 B 2025.09.02D13:36:03.880017270 4 PRIMARY AUTOFOVFEED
200 kxsREPL_A1 | A REPL A 2025.09.03D12:30:55.020216449 4 SECONDARY AUTOFOVFEED
200 kxsREPL_B1 | B REPL A 2025.09.03D12:30:55.020216449 4 PRIMARY AUTOFOVFEED
Use the state command¶
You can look up the state of feeds as reported by one or more nodes in your environment by means of the state command. For each node, it shows the same data as the sysstate command as well as detailed health metrics on each feed such as the last watermark published, the number of aged message segments, the timestamp of the last message received, the timestamp of the last heartbeat, etc.
The state command is based on information from discovery, merged with real-time heartbeats sent from each active HA process in the cluster.
| Syntax | `state;`<nodes> |
|---|---|
| Optional | nodes |
| Arming required | No |
| Confirmation required | No |
q).maint.ha[`state]
feed node| feedName sndNode ts cc mode event lastWm ..
---------| ----------------------------------------------------------------------|.
0 A | MD B 2025.08.30D02:13:02.617429167 4 SECONDARY AUTOFOVFEED (390462;0..
0 B | MD B 2025.08.30D02:13:02.617429167 4 PRIMARY AUTOFOVFEED (390462;0..
100 A | feed1 B 2025.09.02D13:36:03.880017270 4 SECONDARY AUTOFOVFEED (405354;0..
100 B | feed1 B 2025.09.02D13:36:03.880017270 4 PRIMARY AUTOFOVFEED (405354;0..
200 A | REPL A 2025.09.03D12:30:55.020216449 4 SECONDARY AUTOFOVFEED ()
200 B | REPL A 2025.09.03D12:30:55.020216449 4 PRIMARY AUTOFOVFEED ()
The following columns give important information about the health of each feed. These columns will be blank if there are no active feeds.
| Name | Description |
|---|---|
| lastWm | Last watermark that was published. A watermark is a computed value that identifies the level of progress of a node. It is sent from the primary to secondaries and secondaries use it to evaluate their state as compared with the primary. The watermark is composed of a timestamp and an opaque quantity called a fingerprint. The fingerprint format is arbitrary and feed-specific, and it may consist of multiple properties that collectively provide the required uniqueness of the underlying message to which it refers. |
| lastWmTS | Timestamp of the last watermark |
| lastPos | Long position of the last processed message in the stream |
| lastTS | Timestamp of last message received |
| emsTS | Timestamp of EMS assessment at last nominal transition |
| isQr | Indicates whether or not any node has quorum for a feed |
| isDelinq | Indicates whether or not any node has been delinquent in reporting to the associated node. 0 = all expected nodes have reported to the associated node 1 = at least one expected node has failed to report to the associated node |
| bp | Indicates whether or not the node has a busy operation pending for the feed. 1 = the node is primary and has deferred an operation such as garbage collection until it becomes secondary, and is offering the feed to a healthy secondary. 0 = no operation pending |
| rcvTS | Timestamp when the last heartbeat was received |
| prevTS | Timestamp when the previous heartbeat was received |
Look up your feed configuration¶
You can look up your feed configuration by means of the feeds command. For each feed, it shows the name, the feed ID, the feed type, the assigned primary and secondary nodes and the failover nodes. There are four possible feed types (or flags):
| Flag | Feed type |
|---|---|
| A | Acting Secondary |
| C | Coherent |
| M | Mediated |
| Syntax | `feeds;`<feeds> |
|---|---|
| Optional | feeds |
| Arming required | No |
q).maint.ha[`feeds]
name | id flags nodes failoverNodes
-----|-----------------------------
MD | 0 MA A B A B
REPL | 200 MA A B A B
feed1| 100 MA A B A B