Events

Event Types

The OpenSVC daemon generates two types of events.

Event kind "patch"

Changes in the cluster monitors data, presented as json patches.

Those are generated upon merging cluster nodes data. They are the most frequent kind of payload exchanged through the heartbeats.

Example:

{
  "nodename": "aubergine",                            # cluster node
  "kind": "patch",
  "data": [
    [
      ["services", "status", "tstscaler", "monitor"], # key
      {                                               # new value
        "status": "scaling",
        "status_updated": 1539074932.4393582,
        "global_expect_updated": 1539074931.857869,
        "global_expect": null,
        "placement": "leader"
      }
    ]
  ]
}

Event kind "event"

These events are not cluster-wide. They are generated by the daemon threads on critical state changes and orchestration decisions on its local objects.

These events have a dictionnary in the "data" key, with the following sub-keys:

  • id: The event id
  • reason: An event id can be triggered for different reasons, in which case the reason key might be provided to explain the situation.
  • svcname: if specified, the event concerns a service, in which case a snapshot of the service data is also provided in the "service" and "instance" keys.
  • monitor: a snapshot of the node monitor states

Example:

{
  "nodename": "aubergine",
  "kind": "event",
  "data": {
    "id": "instance_thaw",                       # event id
    "reason": "target",                          # event reason
    "svcname": "ha1",
    "monitor": {                                 # node monitor states
      "status": "idle",
      "status_updated": 1539074255.1265483
    },
    "service": {                                 # service aggregated states
      "avail": "up",
      "frozen": "frozen",
      "overall": "warn",
      "placement": "optimal",
      "provisioned": true
    },
    "instance": {                                # service instance states
      "updated": "2018-10-09T08:59:00.317291Z",
      "mtime": 1539075540.317291,
      "app": "default",
      "env": "DEV",
      "placement": "spread",
      "topology": "flex",
      "provisioned": true,
      "running": [],
      "flex_min_nodes": 1,
      "flex_max_nodes": 2,
      "frozen": true,
      "orchestrate": "ha",
      "status_group": {
        "fs": "n/a",
        "ip": "up",
        "task": "n/a",
        "app": "n/a",
        "sync": "n/a",
        "disk": "n/a",
        "container": "n/a",
        "share": "n/a"
      },
      "overall": "warn",
      "avail": "up",
      "optional": "n/a",
      "csum": "95b8b5a953d16be504999612d0159949",
      "monitor": {                               # service instance monitor states
        "status": "idle",
        "status_updated": 1539074254.7616527,
        "global_expect_updated": 1539075568.6204853,
        "local_expect": "started",
        "global_expect": "thawed",
        "placement": ""
      }
    }
  }
}

Hooks

Custom scripts can be executed on events. These hooks are defined in the node configuration file.

Example:

[hook#1]
events = all
command = /root/on_event

Events are specified by id only. The keyword accepts multiple ids formatted as a comma-separated list, or the all special value.

The script referenced by the command keyword can get the whole event data on stdin.

Hooks executions are logged in the node log.

Watching Events

In human readable format:

om node events

In machine readable format:

om node events --format json

Waiting for data change events

# test the filter
$ om daemon status --filter "monitor.nodes.nuc-cva.frozen"
0

# already on target => return immediately
$ om node wait --filter "monitor.nuc-cva.frozen=0" --duration 1s

$ echo $?
0

# not going on target => timeout
$ om node wait --filter "monitor.nuc-cva.frozen" --duration 1s
timeout

$ echo $?
1