Docker High Availability¶
Considering you are responsible of hosting docker production infrastructures, you are surely interested in improving the service availability. Just imagine you have stacked tens of docker environments on one physical host, you certainly want to ensure that all people accessing your platform won't be prevented from working due to a hardware failure.
The first step consists in implementing a manual failover cluster, like described in Docker relocation tutorial.
The second step, shown in this tutorial, will show up how you can followup with a HA configuration by implementing OpenHA on top of OpenSVC services, leading to a dual node high availability docker cluster.
OpenSVC agent, and OpenHA are both open source free softwares.
Pre-requisites¶
OpenSVC services managing docker container(s), in this tutorial, we use only 2 services
- first one is registry.opensvc.com
- second one is busybox.opensvc.com
2 physical servers both having :
- network attachments on the same IP subnet
- shared storage : same block device seen from operating system
- OpenSVC agent installed
- Docker installed
- OpenHA installed (Installation described in Getting Started - OpenHA Integration and High Availability Setup)
- require heartbeat implementation (network, disk, both)
- 2 services configured
Initial State¶
We assume that both OpenSVC services are up and running on node deb1.opensvc.com
Service registry.opensvc.com¶
Service configuration file:
root@deb1:/etc/opensvc# cat registry.opensvc.com.env
[DEFAULT]
autostart_node = deb1.opensvc.com
app = OSVCLAB
service_type = DEV
nodes = deb1.opensvc.com deb2.opensvc.com
docker_data_dir = /opt/registry.opensvc.com/dockerdata
docker_daemon_args = --ip 37.59.71.25
[container#1]
type = docker
run_image = de9a91835513
run_args = -v /opt/registry.opensvc.com/registrydata:/registrydata -e STORAGE_PATH=/registrydata -p 5000:5000
[ip#1]
ipdev = eth0
ipname = registry.opensvc.com
[vg#1]
vgname = vgregistry
[fs#1]
mnt_opt = rw
mnt = /opt/registry.opensvc.com
dev = /dev/mapper/vgregistry-lvregistryroot
type = ext4
[fs#2]
mnt_opt = rw
mnt = /opt/registry.opensvc.com/dockerdata
dev = /dev/mapper/vgregistry-lvdockerdata
type = ext4
[fs#3]
mnt_opt = rw
mnt = /opt/registry.opensvc.com/registrydata
dev = /dev/mapper/vgregistry-lvregistrydata
type = ext4
Service state on node deb1.opensvc.com:
root@deb1:/etc/opensvc# registry.opensvc.com print status
registry.opensvc.com
overall up
|- avail up
| |- container#1 .... up e9828ec4620a@registry:latest
| |- vg#1 .... up vgregistry
| |- fs#1 .... up /dev/mapper/vgregistry-lvregistryroot@/opt/registry.opensvc.com
| |- fs#2 .... up /dev/mapper/vgregistry-lvdockerdata@/opt/registry.opensvc.com/dockerdata
| |- fs#3 .... up /dev/mapper/vgregistry-lvregistrydata@/opt/registry.opensvc.com/registrydata
| '- ip#1 .... up registry.opensvc.com@eth0
|- sync up
| '- sync#i0 .... up rsync svc config to drpnodes, nodes
'- hb n/a
root@deb1:/etc/opensvc# registry.opensvc.com docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e9828ec4620a registry:latest /bin/sh -c 'exec doc 23 minutes ago Up 23 minutes 37.59.71.25:5000->5000/tcp registry.opensvc.com.container.1
Service state on node deb2.opensvc.com:
root@deb2:/etc/opensvc# registry.opensvc.com print status
registry.opensvc.com
overall down
|- avail down
| |- container#1 .... down de9a91835513
| | | # docker daemon is not running
| |- vg#1 .... down vgregistry
| |- fs#1 .... down /dev/mapper/vgregistry-lvregistryroot@/opt/registry.opensvc.com
| |- fs#2 .... down /dev/mapper/vgregistry-lvdockerdata@/opt/registry.opensvc.com/dockerdata
| |- fs#3 .... down /dev/mapper/vgregistry-lvregistrydata@/opt/registry.opensvc.com/registrydata
| '- ip#1 .... down registry.opensvc.com@eth0
|- sync up
| '- sync#i0 .... up rsync svc config to drpnodes, nodes
'- hb n/a
Service busybox.opensvc.com¶
Service configuration file:
root@deb1:/etc/opensvc# cat busybox.opensvc.com.env
[DEFAULT]
autostart_node = deb1.opensvc.com
app = OSVCLAB
service_type = DEV
nodes = deb1.opensvc.com deb2.opensvc.com
docker_data_dir = /opt/busybox.opensvc.com/appdata
docker_daemon_args = --ip 37.59.71.24
[container#1]
type = docker
run_image = b073e328878e
subset = 00database
[container#2]
type = docker
run_image = b073e328878e
subset = 01appservers
[container#3]
type = docker
run_image = b073e328878e
subset = 01appservers
[container#4]
type = docker
run_image = b073e328878e
subset = 01appservers
[container#5]
type = docker
run_image = b073e328878e
subset = 01appservers
[subset#container.docker:01appservers]
parallel = true
[container#6]
type = docker
run_image = b073e328878e
subset = 02webservers
[container#7]
type = docker
run_image = b073e328878e
subset = 02webservers
[subset#container.docker:02webservers]
parallel = false
[ip#1]
ipdev = eth0
ipname = busybox.opensvc.com
[vg#1]
vgname = vgbusybox
scsireserv = false
[fs#1]
mnt_opt = rw
mnt = /opt/busybox.opensvc.com
dev = /dev/mapper/vgbusybox-lvbusyboxroot
type = ext4
[fs#2]
mnt_opt = rw
mnt = /opt/busybox.opensvc.com/appdata
dev = /dev/mapper/vgbusybox-lvbusyboxdata
type = ext4
Service state on deb1.opensvc.com:
root@deb1:/etc/opensvc# busybox.opensvc.com print status
busybox.opensvc.com
overall up
|- avail up
| |- container#1 .... up c37b83887947@opensvc/busybox:date
| |- container#2 .... up a189884f060a@opensvc/busybox:date
| |- container#3 .... up fb2513423499@opensvc/busybox:date
| |- container#4 .... up 3a7e545bced2@opensvc/busybox:date
| |- container#5 .... up 1b6a3d00661e@opensvc/busybox:date
| |- container#6 .... up 173a812f4ed5@opensvc/busybox:date
| |- container#7 .... up 59fca3cd2745@opensvc/busybox:date
| |- vg#1 .... up vgbusybox
| |- fs#1 .... up /dev/mapper/vgbusybox-lvbusyboxroot@/opt/busybox.opensvc.com
| |- fs#2 .... up /dev/mapper/vgbusybox-lvbusyboxdata@/opt/busybox.opensvc.com/appdata
| '- ip#1 .... up busybox.opensvc.com@eth0
|- sync up
| '- sync#i0 .... up rsync svc config to drpnodes, nodes
'- hb n/a
root@deb1:/etc/opensvc# busybox.opensvc.com docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
59fca3cd2745 opensvc/busybox:date /bin/sh -c 'while tr 13 minutes ago Up 13 minutes busybox.opensvc.com.container.7
173a812f4ed5 opensvc/busybox:date /bin/sh -c 'while tr 13 minutes ago Up 13 minutes busybox.opensvc.com.container.6
fb2513423499 opensvc/busybox:date /bin/sh -c 'while tr 13 minutes ago Up 13 minutes busybox.opensvc.com.container.3
3a7e545bced2 opensvc/busybox:date /bin/sh -c 'while tr 13 minutes ago Up 13 minutes busybox.opensvc.com.container.4
a189884f060a opensvc/busybox:date /bin/sh -c 'while tr 13 minutes ago Up 13 minutes busybox.opensvc.com.container.2
1b6a3d00661e opensvc/busybox:date /bin/sh -c 'while tr 13 minutes ago Up 13 minutes busybox.opensvc.com.container.5
c37b83887947 opensvc/busybox:date /bin/sh -c 'while tr 13 minutes ago Up 13 minutes busybox.opensvc.com.container.1
Service state on deb2.opensvc.com:
root@deb2:/etc/opensvc# busybox.opensvc.com print status
busybox.opensvc.com
overall down
|- avail down
| |- container#1 .... down b073e328878e
| | | # docker daemon is not running
| |- container#2 .... down b073e328878e
| | | # docker daemon is not running
| |- container#3 .... down b073e328878e
| | | # docker daemon is not running
| |- container#4 .... down b073e328878e
| | | # docker daemon is not running
| |- container#5 .... down b073e328878e
| | | # docker daemon is not running
| |- container#6 .... down b073e328878e
| | | # docker daemon is not running
| |- container#7 .... down b073e328878e
| | | # docker daemon is not running
| |- vg#1 .... down vgbusybox
| |- fs#1 .... down /dev/mapper/vgbusybox-lvbusyboxroot@/opt/busybox.opensvc.com
| |- fs#2 .... down /dev/mapper/vgbusybox-lvbusyboxdata@/opt/busybox.opensvc.com/appdata
| '- ip#1 .... down busybox.opensvc.com@eth0
|- sync up
| '- sync#i0 .... up rsync svc config to drpnodes, nodes
'- hb n/a
Nodes State¶
From the physical nodes point of view, the services are in the following states (check overall status column):
UP on node deb1.opensvc.com:
root@deb1:/etc/opensvc# svcmon --service=registry.opensvc.com,busybox.opensvc.com
service service container container ip disk fs share app hb sync avail overall
name type type status status status status status status status status status status frozen
------- ------- --------- --------- ------ ------ ------ ------ ------ ------ ------ ------ ------- ------
registry.opensvc.com DEV hosted up up up up n/a n/a n/a up up up False
busybox.opensvc.com DEV hosted up up up up n/a n/a n/a up up up False
DOWN on node deb2.opensvc.com:
root@deb2:/etc/opensvc# svcmon --service=registry.opensvc.com,busybox.opensvc.com
service service container container ip disk fs share app hb sync avail overall
name type type status status status status status status status status status status frozen
------- ------- --------- --------- ------ ------ ------ ------ ------ ------ ------ ------ ------- ------
registry.opensvc.com DEV hosted down down down down n/a n/a n/a up down down False
busybox.opensvc.com DEV hosted down down down down n/a n/a n/a up down down False
Services registry.opensvc.com and busybox.opensvc.com are currently running on node deb1.opensvc.com. We know that they can be manually relocated on node deb2.opensvc.com, and we need to do it automatically.
Cluster Configuration¶
One important concept to understand is that the HA cluster is made with 2 different pieces of software
- OpenSVC agent : deals with resources management, and service actions, actually acts as a cluster resource manager
- this topic is already implemented, described in the previous tutorial part
- OpenHA : used to monitor both cluster nodes, and take decisions about where to start/stop/failover which service.
- this subject is covered in the following section
OpenSVC Configuration¶
2 modifications are expected:
- remove the autostart_node keyword from service configuration file
- OpenSVC is no more deciding where the service start, it is now the OpenHA decision.
- tell OpenSVC that it now depends on OpenHA
First modification implementation:
root@deb1:/# cd /etc/opensvc
root@deb1:/etc/opensvc# cp registry.opensvc.com.env registry.opensvc.com.backup
root@deb1:/etc/opensvc# cat registry.opensvc.com.backup | grep -v autostart_node > registry.opensvc.com.env
root@deb1:/etc/opensvc# diff registry.opensvc.com.backup registry.opensvc.com.env
2d1
< autostart_node = deb1.opensvc.com
root@deb1:/etc/opensvc# cp busybox.opensvc.com.env busybox.opensvc.com.backup
root@deb1:/etc/opensvc# cat busybox.opensvc.com.backup | grep -v autostart_node > busybox.opensvc.com.env
root@deb1:/etc/opensvc# diff busybox.opensvc.com.backup busybox.opensvc.com.env
2d1
< autostart_node = deb1.opensvc.com
Second modification implementation:
root@deb1:/etc/opensvc# cat >> registry.opensvc.com.env << EOF
[hb#0]
type = OpenHA
name = registry
EOF
root@deb1:/etc/opensvc# cat >> busybox.opensvc.com.env << EOF
[hb#0]
type = OpenHA
name = busybox
EOF
root@deb1:/etc/opensvc# tail -3 registry.opensvc.com.env
[hb#0]
type = OpenHA
name = registry
root@deb1:/etc/opensvc# tail -3 busybox.opensvc.com.env
[hb#0]
type = OpenHA
name = busybox
Propagate configuration:
root@deb1:/etc/opensvc# allupservices syncnodes --force
Note
this is a mass action command. Each service in "up" state will propagate its configuration to other nodes. We could have just required the push for our 2 services only.
To continue, we stop both services (not mandatory, but easier to understand in this tutorial context)
on node deb1.opensvc.com:
root@deb1:/etc/opensvc# registry.opensvc.com stop --cluster
root@deb1:/etc/opensvc# busybox.opensvc.com stop --cluster
Warning
as OpenSVC is no more supposed to pilot services itself, due to the declared heartbeat resource in the <service>.env file, a security prevents it to stop service. It can be overridden by the --cluster option.
OpenHA installation¶
Installation described in Getting Started - OpenHA Integration and High Availability Setup)
# wget -O /tmp/openha.latest.rpm http://repo.opensvc.com/rpms/deps/el6/openha-0.4.1.osvc1-0.x86_64.rpm
# alien --to-deb /tmp/openha.latest.rpm
# dpkg -i openha_0.4.1.osvc1-1_amd64.deb
# dpkg -l|grep -i openha
ii openha 0.4.1.osvc1-1 amd64 EASY high-availability software.
# ls /usr/local/cluster
bin conf doc env.sh ezha.init log services
OpenHA configuration¶
In file /usr/local/cluster/conf/nodes we list physical cluster nodes.
On both nodes:
# cat /usr/local/cluster/conf/nodes
deb1.opensvc.com
deb2.opensvc.com
The heartbeat configuration file /usr/local/cluster/conf/monitor contains the following lines.
Note
Whatever the heartbeat type (network, disk), the same logic is used, the first node announce its heartbeat status while the other node reads it.
On both nodes:
# cat /usr/local/cluster/conf/monitor
deb1.opensvc.com net eth1 239.131.50.10 5010 10
deb1.opensvc.com net eth2 239.131.50.11 5011 10
deb2.opensvc.com net eth1 239.131.50.12 5012 10
deb2.opensvc.com net eth2 239.131.50.13 5013 10
deb1.opensvc.com dio /dev/mapper/disk.hb.0 0 10
deb2.opensvc.com dio /dev/mapper/disk.hb.0 10 10
This 3 heartbeats (2 networks + 1 disk) configuration means:
deb1.opensvc.comnode will send heartbeat through eth1 on multicast IP 239.131.50.10 port 5010, with a 10 seconds timeout, anddeb2.opensvc.comnode will listen on the same ip portdeb1.opensvc.comnode will send heartbeat through eth2 on multicast IP 239.131.50.11 port 5011, with a 10 seconds timeout, anddeb2.opensvc.comnode will listen on the same ip portdeb2.opensvc.comnode will send heartbeat through eth1 on multicast IP 239.131.50.12 port 5012, with a 10 seconds timeout, anddeb1.opensvc.comnode will listen on the same ip portdeb2.opensvc.comnode will send heartbeat through eth2 on multicast IP 239.131.50.13 port 5013, with a 10 seconds timeout, anddeb1.opensvc.comnode will listen on the same ip portdeb1.opensvc.comnode will write heartbeat starting from the first block of disk/dev/mapper/disk.hb.0with a 10 seconds timeout, anddeb2.opensvc.comnode will read it from the first blockdeb2.opensvc.comnode will write heartbeat starting from block number 10 of disk/dev/mapper/disk.hb.0with a 10 seconds timeout, anddeb1.opensvc.comnode will read it from the block number 10
Note
adjust heartbeat configuration according to your setup. Only disk, only network, a mix of both.
The OpenHA services are configured and mapped 1 to 1 to OpenSVC services:
Loading cluster environment, on both nodes:
# export EZ=/usr/local/cluster
# . $EZ/env.sh
Creating OpenHA service registry, on both nodes:
# $EZ_BIN/service -a registry /etc/opensvc/registry.opensvc.com.cluster deb1.opensvc.com deb2.opensvc.com /bin/true
Creating service registry :
Directory /usr/local/cluster/services/registry created
State file /usr/local/cluster/services/registry/STATE.deb1.opensvc.com created with initial state FROZEN_STOP
State file /usr/local/cluster/services/registry/STATE.deb2.opensvc.com created with initial state FROZEN_STOP
Done.
Creating OpenHA service busybox, on both nodes:
# $EZ_BIN/service -a busybox /etc/opensvc/busybox.opensvc.com.cluster deb1.opensvc.com deb2.opensvc.com /bin/true
Creating service busybox :
Directory /usr/local/cluster/services/busybox created
State file /usr/local/cluster/services/busybox/STATE.deb1.opensvc.com created with initial state FROZEN_STOP
State file /usr/local/cluster/services/busybox/STATE.deb2.opensvc.com created with initial state FROZEN_STOP
Done.
- -a busybox : add an OpenHA service named busybox
- /etc/opensvc/busybox.opensvc.com.cluster : is the start/stop script for the service
- deb1.opensvc.com : is the primary node
- deb2.opensvc.com : is the secondary node
- /bin/true : is the check-up script ran before service startup. Should be replaced by a stonith command for production implementation.
OpenHA Start¶
on node deb1.opensvc.com:
root@deb1:~# $EZ/ezha start
root@deb1:~# ez_ps
root 25004 0.0 0.1 16664 924 pts/10 S< 07:46 0:00 /usr/local/cluster/bin/heartd eth1 239.131.50.10 5010
root 25007 0.0 0.1 16664 920 pts/10 S< 07:46 0:00 /usr/local/cluster/bin/heartd eth2 239.131.50.11 5011
root 25010 0.0 0.0 16452 500 pts/10 S< 07:46 0:00 /usr/local/cluster/bin/heartc eth1 239.131.50.12 5012 10
root 25013 0.0 0.0 16452 500 pts/10 S< 07:46 0:00 /usr/local/cluster/bin/heartc eth2 239.131.50.13 5013 10
root 25016 0.0 0.1 16452 908 pts/10 S< 07:46 0:00 /usr/local/cluster/bin/heartd_dio /dev/mapper/disk.hb.0 0
root 25019 0.0 0.1 16240 756 pts/10 S< 07:46 0:00 /usr/local/cluster/bin/heartc_dio /dev/mapper/disk.hb.0 10 10
root 25024 0.0 0.2 97928 1052 pts/10 S 07:46 0:00 /usr/local/cluster/bin/nmond
root@deb1:~# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
registry deb1.opensvc.com FROZEN_STOP deb2.opensvc.com UNKNOWN
busybox deb1.opensvc.com FROZEN_STOP deb2.opensvc.com UNKNOWN
Note
FROZEN_STOP is the expected, default OpenHA service status at first service declaration
root@deb1:~# ez_hb_status
=> Querying OpenHA heartbeat status [/usr/local/cluster/bin/hb -s]
net sender eth1:239.131.50.10 pid 25004 status UP, updated at Jun 19 08:00:57
net sender eth2:239.131.50.11 pid 25007 status UP, updated at Jun 19 08:00:57
net listener eth1:239.131.50.12 status DOWN
net listener eth2:239.131.50.13 status DOWN
dio sender /dev/mapper/disk.hb.0:0 pid 25016 status UP, updated at Jun 19 08:00:58
dio listener /dev/mapper/disk.hb.0:10 status DOWN
Note
although deb1 heartbeat senders are announcing their state accurately, the listeners processes do not manage to read anything announced from deb2, it is a normal state because OpenHA is not started on deb2 at the moment
on node deb2.opensvc.com:
root@deb2:/# $EZ/ezha start
root@deb2:/# ez_ps
root 12407 0.0 0.1 16452 736 pts/0 S< 08:10 0:00 /usr/local/cluster/bin/heartc eth1 239.131.50.10 5010 10
root 12410 0.0 0.1 16452 740 pts/0 S< 08:10 0:00 /usr/local/cluster/bin/heartc eth2 239.131.50.11 5011 10
root 12413 0.0 0.1 16664 920 pts/0 S< 08:10 0:00 /usr/local/cluster/bin/heartd eth1 239.131.50.12 5012
root 12416 0.0 0.1 16664 920 pts/0 S< 08:10 0:00 /usr/local/cluster/bin/heartd eth2 239.131.50.13 5013
root 12419 0.0 0.1 16240 768 pts/0 S< 08:10 0:00 /usr/local/cluster/bin/heartc_dio /dev/mapper/disk.hb.0 0 10
root 12422 0.0 0.1 16452 908 pts/0 S< 08:10 0:00 /usr/local/cluster/bin/heartd_dio /dev/mapper/disk.hb.0 10
root 12426 0.0 0.2 32392 1040 pts/0 S 08:10 0:00 /usr/local/cluster/bin/nmond
root@deb2:/# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
busybox deb1.opensvc.com FROZEN_STOP deb2.opensvc.com FROZEN_STOP
registry deb1.opensvc.com FROZEN_STOP deb2.opensvc.com FROZEN_STOP
root@deb2:/# ez_hb_status
=> Querying OpenHA heartbeat status [/usr/local/cluster/bin/hb -s]
net listener eth1:239.131.50.10 pid 25004 status UP, updated at Jun 19 08:10:49
net listener eth2:239.131.50.11 pid 25007 status UP, updated at Jun 19 08:10:49
net sender eth1:239.131.50.12 pid 12413 status UP, updated at Jun 19 08:10:49
net sender eth2:239.131.50.13 pid 12416 status UP, updated at Jun 19 08:10:49
dio listener /dev/mapper/disk.hb.0:0 pid 25016 status UP, updated at Jun 19 08:10:40
dio sender /dev/mapper/disk.hb.0:10 pid 12422 status UP, updated at Jun 19 08:10:49
Note
all heartbeat processes are OK, senders and listeners. It is the same on the other node deb1 (you can check)
OpenHA Services¶
As you have seen, both OpenHA service are in FROZEN_STOP state, which means that they are declared in OpenHA, but their state can't change. They first have to be unfrozen.
check OpenSVC state, service should be stopped on both nodes:
root@deb1:/# svcmon --service=registry.opensvc.com,busybox.opensvc.com
service service container container ip disk fs share app hb sync avail overall
name type type status status status status status status status status status status frozen
------- ------- --------- --------- ------ ------ ------ ------ ------ ------ ------ ------ ------- ------
registry.opensvc.com DEV hosted down down down down n/a n/a warn up down warn False
busybox.opensvc.com DEV hosted down down down down n/a n/a warn up down warn False
root@deb2:/# svcmon --service=registry.opensvc.com,busybox.opensvc.com
service service container container ip disk fs share app hb sync avail overall
name type type status status status status status status status status status status frozen
------- ------- --------- --------- ------ ------ ------ ------ ------ ------ ------ ------ ------- ------
registry.opensvc.com DEV hosted down down down down n/a n/a warn up down warn False
busybox.opensvc.com DEV hosted down down down down n/a n/a warn up down warn False
Unfreeze service registry on node deb1.opensvc.com:
root@deb1:/# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
registry deb1.opensvc.com FROZEN_STOP deb2.opensvc.com FROZEN_STOP
busybox deb1.opensvc.com FROZEN_STOP deb2.opensvc.com FROZEN_STOP
root@deb1:/# $EZ_BIN/service -A registry unfreeze
Ready to UNFREEZE, we are (FROZEN_STOP)
Service registry UNFROZEN
root@deb1:/# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
registry deb1.opensvc.com START_READY deb2.opensvc.com FROZEN_STOP
busybox deb1.opensvc.com FROZEN_STOP deb2.opensvc.com FROZEN_STOP
root@deb1:/# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
registry deb1.opensvc.com STARTING deb2.opensvc.com FROZEN_STOP
busybox deb1.opensvc.com FROZEN_STOP deb2.opensvc.com FROZEN_STOP
root@deb1:/# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
registry deb1.opensvc.com STARTED deb2.opensvc.com FROZEN_STOP
busybox deb1.opensvc.com FROZEN_STOP deb2.opensvc.com FROZEN_STOP
root@deb1:/# svcmon --service=registry.opensvc.com,busybox.opensvc.com
service service container container ip disk fs share app hb sync avail overall
name type type status status status status status status status status status status frozen
------- ------- --------- --------- ------ ------ ------ ------ ------ ------ ------ ------ ------- ------
registry.opensvc.com DEV hosted up up up up n/a n/a up up up up False
busybox.opensvc.com DEV hosted down down down down n/a n/a warn up down warn False
root@deb1:/# registry.opensvc.com print status
registry.opensvc.com
overall up
|- avail up
| |- container#1 .... up e9828ec4620a@registry:latest
| |- vg#1 .... up vgregistry
| |- fs#1 .... up /dev/mapper/vgregistry-lvregistryroot@/opt/registry.opensvc.com
| |- fs#2 .... up /dev/mapper/vgregistry-lvdockerdata@/opt/registry.opensvc.com/dockerdata
| |- fs#3 .... up /dev/mapper/vgregistry-lvregistrydata@/opt/registry.opensvc.com/registrydata
| '- ip#1 .... up registry.opensvc.com@eth0
|- sync up
| '- sync#i0 .... up rsync svc config to drpnodes, nodes
'- hb up
'- hb#0 .... up hb.openha
root@deb1:/# registry.opensvc.com docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e9828ec4620a registry:latest /bin/sh -c 'exec doc 47 hours ago Up 2 minutes 37.59.71.25:5000->5000/tcp registry.opensvc.com.container.1
Note
after unfreeze, we see different state changes, FROZEN_STOP=>START_READY=>STARTING=>STARTED. Once STARTED is achieved, the service is up and running
Unfreeze service registry on node deb2.opensvc.com:
root@deb2:/# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
busybox deb1.opensvc.com FROZEN_STOP deb2.opensvc.com FROZEN_STOP
registry deb1.opensvc.com STARTED deb2.opensvc.com FROZEN_STOP
root@deb2:/# $EZ_BIN/service -A registry unfreeze
Ready to UNFREEZE, we are (FROZEN_STOP)
Service registry UNFROZEN
root@deb2:/# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
busybox deb1.opensvc.com FROZEN_STOP deb2.opensvc.com FROZEN_STOP
registry deb1.opensvc.com STARTED deb2.opensvc.com STOPPED
root@deb2:/# svcmon --service=registry.opensvc.com,busybox.opensvc.com
service service container container ip disk fs share app hb sync avail overall
name type type status status status status status status status status status status frozen
------- ------- --------- --------- ------ ------ ------ ------ ------ ------ ------ ------ ------- ------
registry.opensvc.com DEV hosted down down down down n/a n/a down up down down False
busybox.opensvc.com DEV hosted down down down down n/a n/a warn up down warn False
Note
OpenHA service registry is now STARTED@deb1 and STOPPED@deb2.
We can enable the second service busybox, using the same procedure:
Unfreeze service busybox on node deb1.opensvc.com:
root@deb1:/# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
registry deb1.opensvc.com STARTED deb2.opensvc.com STOPPED
busybox deb1.opensvc.com FROZEN_STOP deb2.opensvc.com FROZEN_STOP
root@deb1:/# $EZ_BIN/service -A busybox unfreeze
Ready to UNFREEZE, we are (FROZEN_STOP)
Service busybox UNFROZEN
root@deb1:/# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
registry deb1.opensvc.com STARTED deb2.opensvc.com STOPPED
busybox deb1.opensvc.com START_READY deb2.opensvc.com FROZEN_STOP
root@deb1:/# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
registry deb1.opensvc.com STARTED deb2.opensvc.com STOPPED
busybox deb1.opensvc.com STARTING deb2.opensvc.com FROZEN_STOP
root@deb1:/# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
registry deb1.opensvc.com STARTED deb2.opensvc.com STOPPED
busybox deb1.opensvc.com STARTED deb2.opensvc.com FROZEN_STOP
root@deb1:/# svcmon --service=registry.opensvc.com,busybox.opensvc.com
service service container container ip disk fs share app hb sync avail overall
name type type status status status status status status status status status status frozen
------- ------- --------- --------- ------ ------ ------ ------ ------ ------ ------ ------ ------- ------
registry.opensvc.com DEV hosted up up up up n/a n/a up up up up False
busybox.opensvc.com DEV hosted up up up up n/a n/a up up up up False
root@deb1:/# busybox.opensvc.com print status
busybox.opensvc.com
overall up
|- avail up
| |- container#1 .... up c37b83887947@opensvc/busybox:date
| |- container#2 .... up a189884f060a@opensvc/busybox:date
| |- container#3 .... up fb2513423499@opensvc/busybox:date
| |- container#4 .... up 3a7e545bced2@opensvc/busybox:date
| |- container#5 .... up 1b6a3d00661e@opensvc/busybox:date
| |- container#6 .... up 173a812f4ed5@opensvc/busybox:date
| |- container#7 .... up 59fca3cd2745@opensvc/busybox:date
| |- vg#1 .... up vgbusybox
| |- fs#1 .... up /dev/mapper/vgbusybox-lvbusyboxroot@/opt/busybox.opensvc.com
| |- fs#2 .... up /dev/mapper/vgbusybox-lvbusyboxdata@/opt/busybox.opensvc.com/appdata
| '- ip#1 .... up busybox.opensvc.com@eth0
|- sync up
| '- sync#i0 .... up rsync svc config to drpnodes, nodes
'- hb up
'- hb#0 .... up hb.openha
root@deb1:/# busybox.opensvc.com docker attach 59fca3cd2745
Thu Jun 19 07:07:27 UTC 2014
Thu Jun 19 07:07:28 UTC 2014
Thu Jun 19 07:07:29 UTC 2014
Thu Jun 19 07:07:30 UTC 2014
root@deb1:/# busybox.opensvc.com docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
59fca3cd2745 opensvc/busybox:date /bin/sh -c 'while tr 47 hours ago Up About a minute busybox.opensvc.com.container.7
173a812f4ed5 opensvc/busybox:date /bin/sh -c 'while tr 47 hours ago Up About a minute busybox.opensvc.com.container.6
fb2513423499 opensvc/busybox:date /bin/sh -c 'while tr 47 hours ago Up About a minute busybox.opensvc.com.container.3
3a7e545bced2 opensvc/busybox:date /bin/sh -c 'while tr 47 hours ago Up About a minute busybox.opensvc.com.container.4
a189884f060a opensvc/busybox:date /bin/sh -c 'while tr 47 hours ago Up About a minute busybox.opensvc.com.container.2
1b6a3d00661e opensvc/busybox:date /bin/sh -c 'while tr 47 hours ago Up About a minute busybox.opensvc.com.container.5
c37b83887947 opensvc/busybox:date /bin/sh -c 'while tr 47 hours ago Up About a minute busybox.opensvc.com.container.1
Unfreeze service busybox on node deb2.opensvc.com:
root@deb2:/# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
busybox deb1.opensvc.com STARTED deb2.opensvc.com FROZEN_STOP
registry deb1.opensvc.com STARTED deb2.opensvc.com STOPPED
root@deb2:/# $EZ_BIN/service -A busybox unfreeze
Ready to UNFREEZE, we are (FROZEN_STOP)
Service busybox UNFROZEN
root@deb2:/# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
busybox deb1.opensvc.com STARTED deb2.opensvc.com STOPPED
registry deb1.opensvc.com STARTED deb2.opensvc.com STOPPED
Cluster Failover¶
Two kind of events can trigger a cluster failover:
- all heartbeat listeners do not receive peer notifications for timeout period (which is 10 seconds in our example)
- one monitored ressource is not in the "up" status
- this case is not described here, but in High Availability Resource Monitoring
We will halt node deb1.opensvc.com and observe what happens from deb2.opensvc.com:
- primary node halt is launched around Jun 19 09:31:57
- secondary node is running an infinite loop
while true; do echo "-----------------------------"; ez_svc_status; ez_hb_status; date; sleep 5; done, so as to fully understand what happens
On node deb2.opensvc.com:
-----------------------------
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
busybox deb1.opensvc.com STARTED deb2.opensvc.com STOPPED
registry deb1.opensvc.com STARTED deb2.opensvc.com STOPPED
=> Querying OpenHA heartbeat status [/usr/local/cluster/bin/hb -s]
net listener eth1:239.131.50.10 pid 2861 status UP, updated at Jun 19 09:31:55
net listener eth2:239.131.50.11 pid 2864 status UP, updated at Jun 19 09:31:55
net sender eth1:239.131.50.12 pid 17469 status UP, updated at Jun 19 09:31:55
net sender eth2:239.131.50.13 pid 17472 status UP, updated at Jun 19 09:31:55
dio listener /dev/mapper/disk.hb.0:0 pid 2873 status UP, updated at Jun 19 09:31:48
dio sender /dev/mapper/disk.hb.0:10 pid 17478 status UP, updated at Jun 19 09:31:55
Thu Jun 19 09:31:56 CEST 2014
Note
halt is launched just after this sample
-----------------------------
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
busybox deb1.opensvc.com STARTED deb2.opensvc.com STOPPED
registry deb1.opensvc.com STARTED deb2.opensvc.com STOPPED
=> Querying OpenHA heartbeat status [/usr/local/cluster/bin/hb -s]
net listener eth1:239.131.50.10 pid 2861 status UP, updated at Jun 19 09:31:57
net listener eth2:239.131.50.11 pid 2864 status UP, updated at Jun 19 09:31:57
net sender eth1:239.131.50.12 pid 17469 status UP, updated at Jun 19 09:32:00
net sender eth2:239.131.50.13 pid 17472 status UP, updated at Jun 19 09:32:00
dio listener /dev/mapper/disk.hb.0:0 pid 2873 status UP, updated at Jun 19 09:31:58
dio sender /dev/mapper/disk.hb.0:10 pid 17478 status UP, updated at Jun 19 09:32:00
Thu Jun 19 09:32:01 CEST 2014
Note
we can observe that listeners update time is now fixed to 09:31:57. This means that other node is halted, and no more sending heartbeats.
-----------------------------
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
busybox deb1.opensvc.com STARTED deb2.opensvc.com STOPPED
registry deb1.opensvc.com STARTED deb2.opensvc.com STOPPED
=> Querying OpenHA heartbeat status [/usr/local/cluster/bin/hb -s]
net listener eth1:239.131.50.10 pid 2861 status UP, updated at Jun 19 09:31:57
net listener eth2:239.131.50.11 pid 2864 status UP, updated at Jun 19 09:31:57
net sender eth1:239.131.50.12 pid 17469 status UP, updated at Jun 19 09:32:05
net sender eth2:239.131.50.13 pid 17472 status UP, updated at Jun 19 09:32:05
dio listener /dev/mapper/disk.hb.0:0 pid 2873 status UP, updated at Jun 19 09:31:58
dio sender /dev/mapper/disk.hb.0:10 pid 17478 status UP, updated at Jun 19 09:32:05
Thu Jun 19 09:32:06 CEST 2014
Note
node deb2.opensvc.com is still waiting for timeout expiring, which should occur at around 09:32:07
-----------------------------
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
busybox deb1.opensvc.com UNKNOWN deb2.opensvc.com START_READY
registry deb1.opensvc.com UNKNOWN deb2.opensvc.com START_READY
=> Querying OpenHA heartbeat status [/usr/local/cluster/bin/hb -s]
net listener eth1:239.131.50.10 status DOWN
net listener eth2:239.131.50.11 status DOWN
net sender eth1:239.131.50.12 pid 17469 status UP, updated at Jun 19 09:32:10
net sender eth2:239.131.50.13 pid 17472 status UP, updated at Jun 19 09:32:10
dio listener /dev/mapper/disk.hb.0:0 status DOWN
dio sender /dev/mapper/disk.hb.0:10 pid 17478 status UP, updated at Jun 19 09:32:10
Thu Jun 19 09:32:11 CEST 2014
Note
More than 10 seconds are elapsed since deb1.opensvc.com was halted. Node deb2.opensv.com tag its partner as UNKNOWN, for all services, and start service takeover. When cluster is configured with stonith features, it is the moment where other node is killed, so as to avoid split brain cluster.
-----------------------------
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
busybox deb1.opensvc.com UNKNOWN deb2.opensvc.com STARTING
registry deb1.opensvc.com UNKNOWN deb2.opensvc.com STARTED
=> Querying OpenHA heartbeat status [/usr/local/cluster/bin/hb -s]
net listener eth1:239.131.50.10 status DOWN
net listener eth2:239.131.50.11 status DOWN
net sender eth1:239.131.50.12 pid 17469 status UP, updated at Jun 19 09:32:15
net sender eth2:239.131.50.13 pid 17472 status UP, updated at Jun 19 09:32:15
dio listener /dev/mapper/disk.hb.0:0 status DOWN
dio sender /dev/mapper/disk.hb.0:10 pid 17478 status UP, updated at Jun 19 09:32:15
Thu Jun 19 09:32:16 CEST 2014
Note
we can see that service starup is done in parallel in the "secstate" column
-----------------------------
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
busybox deb1.opensvc.com UNKNOWN deb2.opensvc.com STARTING
registry deb1.opensvc.com UNKNOWN deb2.opensvc.com STARTED
=> Querying OpenHA heartbeat status [/usr/local/cluster/bin/hb -s]
net listener eth1:239.131.50.10 status DOWN
net listener eth2:239.131.50.11 status DOWN
net sender eth1:239.131.50.12 pid 17469 status UP, updated at Jun 19 09:32:20
net sender eth2:239.131.50.13 pid 17472 status UP, updated at Jun 19 09:32:20
dio listener /dev/mapper/disk.hb.0:0 status DOWN
dio sender /dev/mapper/disk.hb.0:10 pid 17478 status UP, updated at Jun 19 09:32:20
Thu Jun 19 09:32:21 CEST 2014
-----------------------------
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
busybox deb1.opensvc.com UNKNOWN deb2.opensvc.com STARTING
registry deb1.opensvc.com UNKNOWN deb2.opensvc.com STARTED
=> Querying OpenHA heartbeat status [/usr/local/cluster/bin/hb -s]
net listener eth1:239.131.50.10 status DOWN
net listener eth2:239.131.50.11 status DOWN
net sender eth1:239.131.50.12 pid 17469 status UP, updated at Jun 19 09:32:25
net sender eth2:239.131.50.13 pid 17472 status UP, updated at Jun 19 09:32:25
dio listener /dev/mapper/disk.hb.0:0 status DOWN
dio sender /dev/mapper/disk.hb.0:10 pid 17478 status UP, updated at Jun 19 09:32:25
Thu Jun 19 09:32:26 CEST 2014
Note
service busybox.opensvc.com is longer to start than service registry.opensvc.com
-----------------------------
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
busybox deb1.opensvc.com UNKNOWN deb2.opensvc.com STARTED
registry deb1.opensvc.com UNKNOWN deb2.opensvc.com STARTED
=> Querying OpenHA heartbeat status [/usr/local/cluster/bin/hb -s]
net listener eth1:239.131.50.10 status DOWN
net listener eth2:239.131.50.11 status DOWN
net sender eth1:239.131.50.12 pid 17469 status UP, updated at Jun 19 09:32:30
net sender eth2:239.131.50.13 pid 17472 status UP, updated at Jun 19 09:32:30
dio listener /dev/mapper/disk.hb.0:0 status DOWN
dio sender /dev/mapper/disk.hb.0:10 pid 17478 status UP, updated at Jun 19 09:32:30
Thu Jun 19 09:32:31 CEST 2014
root@deb2:/# svcmon --service=registry.opensvc.com,busybox.opensvc.com
service service container container ip disk fs share app hb sync avail overall
name type type status status status status status status status status status status frozen
------- ------- --------- --------- ------ ------ ------ ------ ------ ------ ------ ------ ------- ------
registry.opensvc.com DEV hosted up up up up n/a n/a up up up up False
busybox.opensvc.com DEV hosted up up up up n/a n/a up up up up False
root@deb2:/# busybox.opensvc.com docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
59fca3cd2745 opensvc/busybox:date /bin/sh -c 'while tr 47 hours ago Up 17 seconds busybox.opensvc.com.container.7
173a812f4ed5 opensvc/busybox:date /bin/sh -c 'while tr 47 hours ago Up 17 seconds busybox.opensvc.com.container.6
fb2513423499 opensvc/busybox:date /bin/sh -c 'while tr 47 hours ago Up 17 seconds busybox.opensvc.com.container.3
3a7e545bced2 opensvc/busybox:date /bin/sh -c 'while tr 47 hours ago Up 17 seconds busybox.opensvc.com.container.4
a189884f060a opensvc/busybox:date /bin/sh -c 'while tr 47 hours ago Up 17 seconds busybox.opensvc.com.container.2
1b6a3d00661e opensvc/busybox:date /bin/sh -c 'while tr 47 hours ago Up 17 seconds busybox.opensvc.com.container.5
c37b83887947 opensvc/busybox:date /bin/sh -c 'while tr 47 hours ago Up 18 seconds busybox.opensvc.com.container.1
root@deb2:/# date
Thu Jun 19 09:32:52 CEST 2014
Note
both services are now up and running on node deb2.opensvc.com
From halt on primary node to service availability on secondary node, we can see a 1 minute delay, mostly due to application startup times.
This setup ensure a robust and easy solution to environments that require high availability. Lots of clusters are currently running in production with OpenSVC & OpenHA.
Cluster Failback¶
When node deb1.opensvc.com is back, it joins the cluster and discover that secondary node deb2.opensvc.com owns services.
root@deb1:~# ez_svc_status
=> Querying OpenHA services status [/usr/local/cluster/bin/service -s -c]
service prinode pristate secnode secstate
registry deb1.opensvc.com STOPPED deb2.opensvc.com STARTED
busybox deb1.opensvc.com STOPPED deb2.opensvc.com STARTED
root@deb1:~# ez_hb_status
=> Querying OpenHA heartbeat status [/usr/local/cluster/bin/hb -s]
net sender eth1:239.131.50.10 pid 2914 status UP, updated at Jun 19 10:03:11
net sender eth2:239.131.50.11 pid 2941 status UP, updated at Jun 19 10:03:11
net listener eth1:239.131.50.12 pid 17469 status UP, updated at Jun 19 10:03:11
net listener eth2:239.131.50.13 pid 17472 status UP, updated at Jun 19 10:03:11
dio sender /dev/mapper/disk.hb.0:0 pid 3015 status UP, updated at Jun 19 10:03:11
dio listener /dev/mapper/disk.hb.0:10 pid 17478 status UP, updated at Jun 19 10:03:09
root@deb1:~# svcmon --service=registry.opensvc.com,busybox.opensvc.com
service service container container ip disk fs share app hb sync avail overall
name type type status status status status status status status status status status frozen
------- ------- --------- --------- ------ ------ ------ ------ ------ ------ ------ ------ ------- ------
registry.opensvc.com DEV hosted down down down down n/a n/a down up down down False
busybox.opensvc.com DEV hosted down down down down n/a n/a down up down down False
Production Use¶
If you plan to use this setup in production, it is strongly recommended to:
- enable scsi reservation on disks (add
scsi_reserve = truein volume group resources for each service). This will prevent any data corruption that would happen if both nodes mounts filesystems at the same time. - implement STONITH as described in High Availability Stonith. As soon as one secondary node has decided to takeover a service, it will stonith the other node to be sure to be the only node to run the OpenSVC service.