Common Properties

All service resources share common properties and behaviours.

Action Requirements

to be completed

Tagging

A resource can be tagged using the keyword tags. The value is a whitespace-separated list of tags. Tag names can be user-defined or hardcoded in the agent.

Custom Tags

Custom tags ease service management on complex configurations, as they can be used in services and resources selector expressions.

Examples:

Resource Tags
app#db database base
app#tomcat1 appsrv base
app#tomcat2 appsrv
app#tomcat3 appsrv
app#nginx1 websrv base
app#nginx2 websrv
# stop resources tagged 'websrv'
$ sudo svcmgr -s <svcname> --tag websrv stop

# stop resources tagged 'websrv' or 'appsrv'
$ sudo svcmgr -s <svcname> --tag websrv,appsrv stop

# stop resources tagged 'websrv' and 'base'
$ sudo svcmgr -s <svcname> --tag websrv+base stop

Special Tags

Some tag names are reserved and have a particular meaning.

noaction

This tag keep the agent from executing state-changing actions on the resource.

The agent is still running the resource status evaluations.

For example, the resource mapping the ip address activated at vm boot by the operating system must be tagged noaction.

encap

This tag assigns the resource to the encapsulated/slave service. The agent on the master-part of the service does not handle such a resource.

svcmgr print status highlights such resources with the E flag.

$ sudo svcmgr -s testzone print status
testzone                           up        
`- sol3.opensvc.com                up         frozen,
   |                                          idle,
   |                                          started
   |- disk#0                 ..../ n/a        testzone.raw0
   |- disk#1                 ..../ up         testzone.raw1
   |- fs#0                   ..... n/a        dir /tmp/share
   |- share#1                ..../ up         nfs:/tmp/share
   `- container#0            ..../ up         testzone
      |- ip#1                ...E/ down       128.0.1.2@lo0/testzone1
      `- app:a1                               //
         |- app#0            ...E/ n/a        true
         |                                    info: not evaluated
         |                                    (instance not up)
         `- app#1            ...E/ n/a        true
                                              info: not evaluated
                                              (instance not up)

See also

Encapsulation

nostatus

This tag prevents the resource status evaluation. The resource status is set to n/a.

dedicated

This tag is by the ip.docker driver only. If set, the physical network interface card is moved to the container network namespace. This NIC is thus reserved, and should not be used by other resources and services.

Scoping

Like any other resource parameter, tags can be scoped.

[ip#1]
type = crossbow
ipname = 128.0.1.2
ipdev = lo0
ipdevext = {svcname}1
netmask = 32
tags = encap
tags@sol1.opensvc.com = encap noaction

See also

Scoping

Subsets

to be completed

Disabled

A resource can be marked as disabled using the disable keyword.

[container#1]
type = docker
run_image = ubuntu:14.04
run_command = /bin/bash
run_args = -i -t
disable = true

This will make the agent ignore any action upon this resource.

svcmgr print status will highlights disabled resources with the D flag.

$ sudo svcmgr -s app1.dev print status --refresh
app1.dev                     up        
`- deb1.opensvc.com          up         idle, started
   |- ip#0             ..... up         192.168.1.1@lo
   `- container#1      .D... n/a        docker container app1.dev.container.1@ubuntu:14.04

Optional

A resource can be marked as optional using the optional keyword.

[app#0]
script = /bin/true
info = true
stop = true
start = true
optional = true

This parameter allow defining non critical resources in the service.

Service actions won't stop on error reported by optional resources.

svcmgr print status will highlights optional resources with the O flag.

$ sudo svcmgr -s redis.acme.com print status
mysvc1.opensvc.com                up        
`- deb1.opensvc.com               up         idle, started
   |- ip#1                  ..... up         128.0.1.124@lo
   |- disk#1                ..... stdby up   loop /opt/disk1.dd
   |- disk#2                ..... stdby up   loop /opt/disk2.dd
   |- disk#3                ..... stdby up   vg vgtest
   |- fs#1                  ..... up         ext4 /dev/vgtest/lvtest1@/opt/avn/lvtest1
   |- fs#2                  ..... up         ext4 /dev/vgtest/lvtest2@/opt/avn/lvtest2
   |- fs#3                  ..... up         ext4 /dev/disk/by-label/testfs@/opt/avn/lvtest3
   |- share#0               ..../ up         nfs:/opt/avn/lvtest3
   |- app#0                 ..O./ n/a        true
   |                                         info: check is not set
   `- sync#i0               ..O./ up         rsync svc config to drpnodes, nodes

Monitoring

A resource can be marked as monitored using the monitor keyword.

[disk#3]
type = vg
name = vgtest
always_on = nodes
monitor = true

It means that this resource is critical for the service availability.

If the resource goes down, then the agent triggers the monitor_action, which may cause a crash or reboot of the node, or stop of the service, to force a failover.

svcmgr print status will highlights monitored resources with the M flag.

$ sudo svcmgr -s redis.acme.com print status
mysvc1.opensvc.com                up        
`- deb1.opensvc.com               up         idle, started
   |- ip#1                  ..... up         128.0.1.124@lo
   |- disk#1                ..... stdby up   loop /opt/disk1.dd
   |- disk#2                ..... stdby up   loop /opt/disk2.dd
   |- disk#3                M.... stdby up   vg vgtest
   |- fs#1                  ..... up         ext4 /dev/vgtest/lvtest1@/opt/avn/lvtest1
   |- fs#2                  ..... up         ext4 /dev/vgtest/lvtest2@/opt/avn/lvtest2
   |- fs#3                  ..... up         ext4 /dev/disk/by-label/testfs@/opt/avn/lvtest3
   |- share#0               ..../ up         nfs:/opt/avn/lvtest3
   |- app#0                 ..O./ n/a        true
   |                                         info: check is not set
   `- sync#i0               ..O./ up         rsync svc config to drpnodes, nodes

Note

  • High Availability topic is described in High Availability setup
  • restart parameter can be combined with monitor setting, as explained below

Automatic Restart

The restart parameter can be set to make the agent daemon monitor restart the resource if it fails:

[app#0]
script = /bin/true
info = true
stop = true
start = true
optional = true
restart = 2

The restart value is the number of times the daemon will attempt to restart the resource before giving up.

If combined with monitor, the agent will try to restart the failed resource before triggering the monitor_action

Standby resources

Some resources must remain up, even when the service instance is stopped.

For example, in a 2-nodes failover service with a fs resource and a sync.rsync resource replicating the fs, the fs resource must be up on the passive node receive the rsync'ed data. If not, the data gets written to the underlying filesystem.

The always_on keyword can be set in these cases:

[disk#3]
type = vg
name = vgtest
always_on = nodes
monitor = true

Possible values are 'nodes', 'drpnodes' or 'nodes drpnodes', or a list of nodes.

Resources tagged with always_on keyword are started on service boot and start actions, and stopped only on service shutdown action.

svcgr print status will display the stdby up status for up always_on resources, and stdby down status for down always_on resources.

# Primary Node
$ sudo svcmgr -s mysvc.acme.com print status
mysvc1.opensvc.com                up        
`- deb1.opensvc.com               up         idle, started
   |- ip#1                  ..... up         128.0.1.124@lo
   |- disk#1                ..... stdby up   loop /opt/disk1.dd
   |- disk#2                ..... stdby up   loop /opt/disk2.dd
   |- disk#3                M.... stdby up   vg vgtest
   |- fs#1                  ..... up         ext4 /dev/vgtest/lvtest1@/opt/avn/lvtest1
   |- fs#2                  ..... up         ext4 /dev/vgtest/lvtest2@/opt/avn/lvtest2
   |- fs#3                  ..... up         ext4 /dev/disk/by-label/testfs@/opt/avn/lvtest3
   |- share#0               ..../ up         nfs:/opt/avn/lvtest3
   |- app#0                 ..O./ n/a        true
   |                                         info: check is not set
   `- sync#i0               ..O./ up         rsync svc config to drpnodes, nodes


# Secondary Node
mysvc1.opensvc.com         
`- deb2.opensvc.com               warn       warn
   |- ip#1                  ..... down       128.0.1.124@lo
   |- disk#1                ..... stdby up   loop /opt/disk1.dd
   |- disk#2                ..... stdby down loop /opt/disk2.dd
   |- disk#3                M.... stdby up   vg vgtest
   |- fs#1                  ..... down       ext4 /dev/vgtest/lvtest1@/opt/avn/lvtest1
   |- fs#2                  ..... down       ext4 /dev/vgtest/lvtest2@/opt/avn/lvtest2
   |- fs#3                  ..... down       ext4 /dev/disk/by-label/testfs@/opt/avn/lvtest3
   |- share#0               ..../ down       nfs:/opt/avn/lvtest3
   |- app#0                 ..O.. n/a        true
   |                                         info: not evaluated (instance not up)
   `- sync#i0               ..O./ up         rsync svc config to drpnodes, nodes

Warning

Don't set shared disk always on. This would cause data corruption.

Triggers

Triggers are scripts or commands executed before or after a resource action.

Syntax

In a service resource configuration section, triggers can be defined using a keyword formatted as

[blocking_]<when>_<action>[@<scope>] = <command>

Blocking

If the blocking_ prefix is set, a non-zero command return code causes the service action to abort on error.

If the blocking_ prefix is not set, a non-zero command return code causes an error log, but the service action continues.

When

Supported values:

  • pre The command is executed before the normal resource driver action.
  • post The command is executed after the normal resource driver action.

Action

Supported values:

  • start
  • stop
  • run
  • provision
  • unprovision
  • sync_nodes
  • sync_drp
  • sync_resync
  • sync_update

Command

The command value supports shell expressions, like &&, ||, ...

Warning

; is interpreted as a comment begin marker if preceded by a whitespace. To use this character as a separator between commands, be sure to not insert a whitespace before. Example: post_start = echo hey; echo ya is valid but post_start = echo hey ; echo ya is not.

Use Cases

  • Inter-service dependencies

    Using a pre_start trigger to implement a wait loop for a tiers service expected state. A blocking_pre_start trigger can be combined with a wait loop timeout to abort the service startup if the required tiers service does not reach the expected state in time.

  • Service provisioning

    Using a post_provision trigger to create directories, files, run configuration management, deploy app, ... just after the filesystems are provisioned and started.

  • Extra start checks

    Using a blocking_pre_start on the first started resource to verify extra conditions to allow service instance startup.

Resource Devices

The agent reports to the collector the base disks list and and size. For each disk, it also reports the size used by each service.

$ sudo nodemgr pushdisks
aubergine                  Size.Used  Vendor  Model                     
`- disks                 
   |- 002538b471bb6f3c     953g               SAMSUNG MZSLW1T0HMLH-000L1
   |  |- testdrbd          3g
   |  |- testmd            991m
   |  |- pridns            6g
   |  |- testmd2           143m
   |  `- aubergine         943g
   |- aubergine.md125      0
   |- aubergine.md127      0
   `- aubergine.md126      0

This feature depends on a device tree build by the agent.

This tree can be display bottom-up:

aubergine                            Type    Size  Pct of Parent
|- loop1                             linear  0     -
|- md127                             linear  0     -
|- md125                             linear  0     -
|- loop6                             linear  0     -
|- loop4                             linear  0     -
|- loop2                             linear  0     -
|- md126                             linear  0     -
|- loop7                             linear  0     -
|- nvme0n1                           linear  953g  -
|  |- nvme0n1p1                      linear  512m  0%
|  `- nvme0n1p2                      linear  953g  99%
|     |- ubuntu--vg-swap_1           linear  15g   1%
|     `- ubuntu--vg-root             linear  915g  96%
|        |- loop13                   linear  10m   0%
|        |- loop12                   linear  50m   0%
|        |- loop11                   linear  50m   0%
|        |  `- md122                 raid1   49m   98%
|        |     `- md123              raid0   97m   197%
|        |- loop10                   linear  50m   0%
|        |  `- md122                 raid1   49m   98%
|        |     `- md123              raid0   97m   197%
|        |- loop14                   linear  143m  0%
|        |- loop9                    linear  50m   0%
|        |  `- md124                 raid1   49m   98%
|        |     `- md123              raid0   97m   197%
|        |- loop8                    linear  50m   0%
|        |  `- md124                 raid1   49m   98%
|        |     `- md123              raid0   97m   197%
|        `- loop0                    linear  3g    0%
|           `- drbd1                         0     0%
|- loop5                             linear  0     -
`- loop3                             linear  0     -

Or top-bottom:

$ sudo nodemgr print devs --reverse
aubergine                            Type    Parent Use  Size  Ratio
|- drbd1                                     -           0     -
|  `- loop0                          linear  0           3g    -
|     `- ubuntu--vg-root             linear  3g          915g  0%
|        `- nvme0n1p2                linear  915g        953g  96%
|           `- nvme0n1               linear  953g        953g  99%
|- md123                             raid0   -           97m   -
|  |- md124                          raid1   48m         49m   97%
|  |  |- loop8                       linear  49m         50m   98%
|  |  |  `- ubuntu--vg-root          linear  50m         915g  0%
|  |  |     `- nvme0n1p2             linear  915g        953g  96%
|  |  |        `- nvme0n1            linear  953g        953g  99%
|  |  `- loop9                       linear  49m         50m   98%
|  |     `- ubuntu--vg-root          linear  50m         915g  0%
|  |        `- nvme0n1p2             linear  915g        953g  96%
|  |           `- nvme0n1            linear  953g        953g  99%
|  `- md122                          raid1   48m         49m   97%
|     |- loop11                      linear  49m         50m   98%
|     |  `- ubuntu--vg-root          linear  50m         915g  0%
|     |     `- nvme0n1p2             linear  915g        953g  96%
|     |        `- nvme0n1            linear  953g        953g  99%
|     `- loop10                      linear  49m         50m   98%
|        `- ubuntu--vg-root          linear  50m         915g  0%
|           `- nvme0n1p2             linear  915g        953g  96%
|              `- nvme0n1            linear  953g        953g  99%
|- md127                             linear  -           0     -
|- md126                             linear  -           0     -
|- md125                             linear  -           0     -
|- ubuntu--vg-swap_1                 linear  -           15g   -
|  `- nvme0n1p2                      linear  15g         953g  1%
|     `- nvme0n1                     linear  953g        953g  99%
|- nvme0n1p1                         linear  -           512m  -
|  `- nvme0n1                        linear  512m        953g  0%
|- loop3                             linear  -           0     -
|- loop2                             linear  -           0     -
|- loop1                             linear  -           0     -
|- loop7                             linear  -           0     -
|- loop6                             linear  -           0     -
|- loop5                             linear  -           0     -
|- loop4                             linear  -           0     -
|- loop13                            linear  -           10m   -
|  `- ubuntu--vg-root                linear  10m         915g  0%
|     `- nvme0n1p2                   linear  915g        953g  96%
|        `- nvme0n1                  linear  953g        953g  99%
|- loop12                            linear  -           50m   -
|  `- ubuntu--vg-root                linear  50m         915g  0%
|     `- nvme0n1p2                   linear  915g        953g  96%
|        `- nvme0n1                  linear  953g        953g  99%
`- loop14                            linear  -           143m  -
   `- ubuntu--vg-root                linear  143m        915g  0%
      `- nvme0n1p2                   linear  915g        953g  96%
         `- nvme0n1                  linear  953g        953g  99%

The --verbose flag adds the known device paths information:

$ sudo nodemgr print devs --reverse --verbose
aubergine                                         Type    Parent Use  Size  Ratio
|- drbd1                                                  -           0     -
|  `- loop0                                       linear  0           3g    -      /dev/disk/by-label/pridns.fs.1
|     |                                                                            /dev/disk/by-uuid/27489964-94c
|     |                                                                            6-4803-a389-c91463da8d3f
|     |                                                                            /dev/loop0
|     `- ubuntu--vg-root                          linear  3g          915g  0%     /dev/disk/by-id/dm-name-ubuntu
|        |                                                                         --vg-root
|        |                                                                         /dev/disk/by-id/dm-uuid-LVM-vz
|        |                                                                         I1exojgdAZhf3X1Vz8A0C1Ne2EN2sr
|        |                                                                         zZlazB8vy5ey8yftklunzMMMUxJwwC
|        |                                                                         ej
|        |                                                                         /dev/disk/by-
|        |                                                                         uuid/3653539e-3299-448e-b80d-
|        |                                                                         576fb6b71b84
|        |                                                                         /dev/mapper/ubuntu--vg-root
|        |                                                                         /dev/ubuntu-vg/root
|        |                                                                         /dev/dm-0
|        `- nvme0n1p2                             linear  915g        953g  96%    /dev/disk/by-id/lvm-pv-uuid-
|           |                                                                      VilFt5-Ne8T-eVxf-QKnX-n0Zc-
|           |                                                                      LIK2-7ct9Kr
|           |                                                                      /dev/disk/by-id/nvme-
|           |                                                                      SAMSUNG_MZSLW1T0HMLH-
|           |                                                                      000L1_S308NX0J403249-part2
|           |                                                                      /dev/disk/by-id/nvme-eui
|           |                                                                      .002538b471bb6f3c-part2
|           |                                                                      /dev/disk/by-partuuid/c6c21095
|           |                                                                      -4a8e-4461-af40-12e78fd758d6
|           |                                                                      /dev/disk/by-path/pci-0000:05:
|           |                                                                      00.0-nvme-1-part2
|           |                                                                      /dev/nvme0n1p2
|           `- nvme0n1                            linear  953g        953g  99%    /dev/disk/by-id/nvme-
...

Each resource driver can implement a method to claim devices as "exposed" and "sub" (exposed layered over). The device tree is walked to determine the "base" devices:

$ sudo svcmgr -s pridns print devs
pridns                     
|- disk#0 (disk.loop)      
|  |- base                 
|  |  `- /dev/nvme0n1
|  `- exposed              
|     `- /dev/loop0
|- fs#1 (fs)               
|  |- base                 
|  |  `- /dev/nvme0n1
|  `- sub                  
|     `- /dev/loop0
`- fs#2 (fs)               
   |- base                 
   |  `- /dev/nvme0n1
   `- sub                  
      `- /dev/loop0

$ sudo svcmgr -s pridns print exposed devs
pridns                     
`- disk#0 (disk.loop)      
   `- exposed              
      `- /dev/loop0

$ sudo svcmgr -s pridns print sub devs
pridns             
|- fs#1 (fs)       
|  `- sub          
|     `- /dev/loop0
`- fs#2 (fs)       
   `- sub          
      `- /dev/loop0

$ sudo svcmgr -s pridns print base devs
pridns                     
|- disk#0 (disk.loop)      
|  `- base                 
|     `- /dev/nvme0n1
|- fs#1 (fs)               
|  `- base                 
|     `- /dev/nvme0n1
`- fs#2 (fs)               
   `- base                 
      `- /dev/nvme0n1