Agent Heartbeats

Hearbeats serve the following purposes:

  • Exchange data between cluster nodes
  • Detect stale nodes
  • Execute the quorum race when a peer becomes stale for its last heartbeat

OpenSVC supports multiple parallel running heartbeats. Exercizing different code paths and infrastructure data paths (network and storage switchs and site interconnects) helps limiting split-brain situation occurences.

Heartbeats are declared in <OSVCETC>/node.conf or <OSVCETC>/cluster.conf, each in a dedicated section named [hb#<n>]. A heartbeat definition should work on all nodes, using scoped keywords if necessary, as the definition are served by the joined node to the joining nodes.

Each heartbeat runs two threads janitored by the agent daemon: tx and rx.

The om daemon status and om mon commands display the heartbeats status, statistics and each peer state

$ sudo svcmon
Threads                                 aubergine clementine nuc
 hb#1.rx    running 224.3.29.71:10001 | /         X          X
 hb#1.tx    running 224.3.29.71:10001 | /         O          O
 hb#2.rx    running 0.0.0.0:10004     | /         X          O
 hb#2.tx    running                   | /         X          O
...

# X: Stale
# O: Sent/Received
# /: Not Applicable

Maintenance

The heartbeat threads are restarted by the agent daemon if they exit uncleanly.

Any command causing a timestamp change on <OSVCETC>/node.conf or <OSVCETC>/cluster.conf triggers heartbeats reconfiguration:

  • Modified parameters are applied
  • Heartbeats removed in the configuration are stopped
  • Heartbeats added in the configuration are started

Stop a heartbeat thread

om daemon stop --thread-id hb#1.tx

Stop the hb#1 tx thread. The thread state transitions from running to stopped.

Start a heartbeat thread

om daemon start --thread-id hb#1.tx

Start the hb#1 tx thread. The thread state transitions from stopped to running.

Set a heartbeat timeout

om cluster set --kw hb#1.timeout=20

Drop a heartbeat

om cluster delete --rid hb#1

Threads

Tx

  • Transmit data prepared by the monitor thread, regularly or as soon as possible after a data change
  • The data is pre-encrypted by the monitor thread

Rx

  • Receive data regularly (disk) or in response to a send (unicast and multicast)
  • Update the received peer data in the cluster data
  • Wake the monitor thread
    • Merge the updated peer data to update the cluster data attributes produced by node data aggregation
    • Run the node orchestrator: handle freeze/thaw requests propagated through the heartbeats
    • Run each service orchestrator: handle service instance start/stop/freeze/thaw transitions, and monitored resources restart
    • Fetch service configuration files from node node with the most recent version
  • If no heartbeat is received from a peer during <hb#n>.timeout, the peer is marked stale in this rx thread. The default timeout is 15 seconds.
  • If all rx threads have a peer marked stale, the peer data is purged from the cluster data
    • immediately if the stale peer has not advertized a maintenance
    • after node.maintenance grace_period if the stale peer has advertized a maintenance

See also

Example Cluster Data

Heartbeat Drivers

Unicast

This driver sends and receives using TCP unicast packets.

[hb#1]
type = unicast

With this simplest definition:

  • The rx thread listens on 0.0.0.0:10000
  • The tx thread sends to <nodename>:10000

A most precise definition would be

[hb#1]
type = unicast
intf@node1 = eth0
intf@node2 = eth2
addr@node1 = 1.2.3.4
addr@node2 = 1.2.3.5
port@node1 = 10001
port@node2 = 10002
timeout = 15

Multicast

This driver sends and receive using UDP multicast packets.

[hb#2]
type = multicast

With this simplest definition:

  • The rx thread listens on all interfaces on port 10000
  • The tx thread sends to 224.3.29.71:10000

A most precise definition would be

[hb#2]
type = multicast
intf@node1 = eth0
intf@node2 = eth2
addr = 224.3.29.71
port = 10001
timeout = 15

Addr and port are not scopable.

Disk

This driver reads and writes on a dedicated disk, using O_DIRECT|O_SYNC|O_DSYNC on a block device on Linux. Other operating systems must use raw char device.

  • The rx thread loops over peer nodes and for each reads its heartbeat data at its reserved slot device offset
  • The tx thread writes to its reserved slot offset on the device
[hb#2]
type = disk
dev = /dev/mapper/3123412312412414214
timeout = 15

When the tx and rx threads are started or reconfigured, they parse a metadata segment at the head of the device and prepare a <nodename>:<slot index> hash.

The metadata zone maximum size is 4MB.

A node metadata slot size is 4k, and contains the cluster node name.

Limits:

  • 1000 nodes (metadata zone size/slot meta data size)
  • nodenames are limited to 4k characters (slot meta data size)
  • A <n>-nodes cluster requires a (<n>+1)*4MB device
  • The heartbeat data (which is gziped) must not exceed 4MB (slot size). A 10 services cluster usually produces ~3k messages.

If a the local nodename is not found in any slot, the thread allocates one.

Relay

This driver reads and writes on a remote opensvc agent memory.

The relay listener <address>:<port> must be reachable from all cluster nodes in normal operations. A relay should be located in a site hosting no other node of the cluster, so this heartbeat can avoid a split when the sites hosting cluster nodes are isolated, but can still reach the relay's site.

The same relay can be used as heartbeat in different clusters. The relay host can also be used as an arbitrator.

  • The rx thread loops over peer nodes and for each requests its heartbeat data from the relay
  • The tx thread sends to the relay
[hb#2]
type = relay
relay = relay1
timeout = 15
secret = 1023102310230123123