Heartbeats

In Kx Control, long-lived connections are maintained between processes and used as a way to determine whether a process is still running. In the case of some failover events (server or network failures), a process can no longer be reachable but the connection isn’t terminated [1]. The client may still send messages across that handle without knowing there is an issue.

To protect against failures and improve recovery time, heartbeats are used between business-critical processes to raise an alert in case one stops responding. Heartbeats can be configured as one- or two-way

One-way heartbeats

In this mode, a client process is configured to send heartbeats to another at a frequency. The server expects to receive heartbeats at that frequency and to take action if a timeout occurs. The timeout value is usually more than 1.5 times the frequency. This mode is used when the server cares about the state of the client but not vice versa.

An example:

  • Feedhandler (FH) is connected to a subscriber
  • If one subscriber goes down, the FH wants to failover and publish to another instead
  • The subscriber doesn’t care if the FH is unresponsive
  • Subscriber acts as the client and initiates heartbeats every 10 seconds
  • FH will close the connection if no heartbeats received within 20 seconds
  • If this occurs, FH knows the subscriber is no longer connected and initiates a failover

Two-way heartbeats

In this mode, both the client and server need to know the state of the other so each is configured to take action if the other fails. The recommended way to configure this is for the client to configure heartbeats but every heartbeat sent should result in an async heartbeat response from the server. Both processes will then use the same frequency and timeout. The set-up is usually configured with the client (heartbeat sender) as the less-critical process so the server doesn’t need to maintain a timer job to send heartbeats [2].

An example:

  • Kx Control launches a process
  • Both need to know if the other is responsive
  • Kx Control needs to know if the process is unresponsive so it can take failover actions (launch process/workflow, send alerts etc)
  • Process needs to know so it can failover to a backup Kx Control process
  • Process acts as the client and sends heartbeats to Kx Control
  • Kx Control echoes those heartbeats back to the process
  • Both processes check for timeouts

Kx Control failover

If Kx Control is installed in a cluster configuration, heartbeats should be configured via the environment variables below. If one server fails, the rest of the cluster will time-out that process and elect a new master.

export DELTACONTROL_HEARTBEAT_TIMEOUT=60
export DELTACONTROL_HEARTBEAT_FREQUENCY=30

Process instances can be configured to heart-beat to Kx Control by setting the heartbeat fields in their reserved parameters. By default these heartbeats will be one-way and Kx Control can be configured to take action when a timeout happens. The action Kx Control takes is configured in the Alerts table of the instance configuration.

Screenshot Screenshot

To enable two-way heartbeats, the CONTROL_HEARTBEATS:<DEFAULT> parameter should be updated. It dictates whether the process should expect responses and what action to take if a failure occurs on the master Kx Control process.

Other components

Some of the Kx Control and Kx Stream frameworks also require protection against failover events. This usually involves setting heartbeats between critical components.

Query Router

The Query Routing (QR) component protects against failures in a couple of ways. If a process doesn’t respond to a query, and a timeout occurs, processes will be pinged or disconnected to ensure they don’t cause continual failures. Between QR processes in a cluster, two-way heartbeats are used to ensure there is always an active master process available to service queries.

Some kdb+ processes can act as clients in the QR framework. If a process registers with heartbeats enabled, these will be two-way between it and the QR, with the client acting as the source. This ensures both sides are aware of any failures of the other. The client will failover to another QR if the master times out.

Messaging

The messaging server (MS) relies on all processes maintaining a long-lived connection in order to publish any subscription changes. Processes connect to the primary on startup and receive any changes as processes come online and register topics. If the MS is unavailable, new subscriptions sent onto a stale handle won’t be acted upon and new processes will connect to the secondary MS. In both cases, existing processes won’t receive subscription updates.

To mitigate this, the MS can be configured with a heartbeat frequency in the Instance Params. If configured, processes will send two-way heartbeats to the server and disconnect it in the event of a failure.

Screenshot

Notes

  1. The connection will eventually close but the length of time this takes depends on the kernel TCP keepalive settings.

  2. This setup could be configured using two sets of one-way heartbeats but this is less efficient as both processes need to run a heartbeat timer job.