Docker Relocation¶
All services are not chaos compliant. For some services it is also preferable not to involve storage and network virtualization nor ingress gateways, to beneficit from the best performance and reliability. For those services a failover OpenSVC service, with or without automatic failover (orchestrate=ha), is a valid solution.
With such services, the running instance can be easily relocated, proactively or in reaction to:
A hardware or software failure involving downtime.
A planned downtime (hardware tech refresh, firmware updates, os upgrades, …)
This relocation capability is more important the higher the number of services.
This tutorial explains how to setup a relocatable dockerized postgresql service, with no network virtualization and a dedicated SAN disk.
Pre-requisites¶
Two nodes with:
network attachments on the same ip subnet, with free ip addresses
shared storage : same block device seen from operating system
OpenSVC agent installed
Docker installed
Service details¶
The service data will be stored in a xfs formatted single logical volume of a volume group build on the shared disk.
The service ip address will be plumbed using ipvlan l2 on a veth given to a google/pause container holding a service-wide netns. This veth will be linked to a node’s bonding interface.
The postgresql database will run in a second container sharing the google/pause container netns.
Create the service¶
$ cat - << EOF | om db1 deploy --config=-
{
"DEFAULT": {
"nodes": "{clusternodes}"
},
"disk#0": {
"name": "{svcname}",
"pvs": "{env.dev}",
"shared": "true",
"type": "vg"
},
"disk#1": {
"name": "data",
"shared": "true",
"size": "100%FREE",
"type": "lv",
"vg": "{svcname}"
},
"fs#1": {
"dev": "/dev/{svcname}/data",
"mnt": "/srv/{svcname}/data",
"shared": "true",
"type": "xfs"
},
"ip#0": {
"gateway": "{env.gateway}",
"ipdev": "{env.intf}",
"ipname": "{env.ipaddr}",
"netmask": "{env.netmask}",
"netns": "container#0",
"mode": "ipvlan-l2",
"type": "netns"
},
"container#0": {
"image": "google/pause",
"rm": "true",
"type": "docker"
},
"container#1": {
"image": "postgres",
"netns": "container#0",
"rm": "true",
"run_args": "-v /srv/{svcname}/data/postgresql:/var/lib/postgresql:rw --user={env.uid}:{env.gid}",
"type": "docker"
},
"env": {
"dev": "/dev/mapper/36589cfc000000e969988ca11344cabb0",
"gateway": "192.168.100.1",
"gid": "999",
"intf": "bond0",
"ipaddr": "192.168.100.14",
"netmask": "24",
"uid": "999"
}
}
EOF
Note
You can change the value of the keywords in the env section using svcmgr --env <kw>=<value> options.
After a few seconds, the service will be up and running on the cluster first node.
$ svcmon -s db1 --sections services
Services node-1-1 node-1-2
db1 up failover | O^ X
Details on node-1-1
$ db1 print status
db1
`- instances
|- node-1-2 down idle
`- node-1-1 up idle, started
|- ip#0 ........ up netns bridge 192.168.100.14/24 br-prd@container#0
|- disk#0 ........ up vg db1
|- disk#1 ........ up lv db1/data
|- fs#1 ........ up xfs /dev/db1/data@/srv/db1/data
|- container#0 ........ up docker container db1.container.0@google/pause
|- container#1 ........ up docker container db1.container.1@postgres
`- sync#i0 ...O./.. up rsync svc config to nodes
Relocate Service¶
$ om db1 switch
node-1-1.db1 service db1 target state set to placed@node-1-2
You can follow the progress of this asynchronous action with svcmon -w -s db1.
After a few seconds, the service is switched
$ svcmon -s db1 --sections services
Services node-1-1 node-1-2
db1 up^ failover | X^ O
Switch logs¶
For reference and timings, here are the logs of this action
$ om db1 logs
# orchestrator logs
20:35:50,004 node-1-1.db1 INFO service db1 target state set to placed@node-1-2
20:35:50,175 node-1-1.db1 INFO stop failover up instance to satisfy the placed@node-1-2 target
# on the source node, stop instance
20:35:50,434 node-1-1.db1 INFO do stop --local (daemon origin)
20:35:50,522 node-1-1.db1.container#1 INFO docker stop b196b8204a5add752680de49f8c9a306888c41db489063455924ada4febb6905
20:35:50,630 node-1-1.db1.container#1 INFO output:
20:35:50,631 node-1-1.db1.container#1 INFO b196b8204a5add752680de49f8c9a306888c41db489063455924ada4febb6905
20:35:50,663 node-1-1.db1.container#1 INFO docker rm db1.container.1
20:35:50,664 node-1-1.db1.container#1 INFO wait for down status
20:35:51,001 node-1-1.db1.ip#0 INFO /usr/bin/nsenter --net=/var/run/docker/netns/21c851c4646b ip addr del 192.168.100.14/24 dev eth0
20:35:51,008 node-1-1.db1.ip#0 INFO /usr/bin/nsenter --net=/var/run/docker/netns/21c851c4646b ip link del dev eth0
20:35:51,023 node-1-1.db1.ip#0 INFO checking 192.168.100.14 availability
20:35:52,210 node-1-1.db1.container#0 INFO docker stop 0363bb5834f23b394065b9a1e31b911e8d848bf01c8e697b55afa926e7849570
20:35:52,305 node-1-1.db1.container#0 INFO output:
20:35:52,307 node-1-1.db1.container#0 INFO 0363bb5834f23b394065b9a1e31b911e8d848bf01c8e697b55afa926e7849570
20:35:52,363 node-1-1.db1.container#0 INFO docker rm db1.container.0
20:35:52,363 node-1-1.db1.container#0 INFO wait for down status
20:35:52,986 node-1-1.db1.fs#1 INFO umount /srv/db1/data
20:35:53,249 node-1-1.db1.disk#1 INFO lvchange -a n db1/data
20:35:53,391 node-1-1.db1.disk#0 INFO vg db1 is already down
# on the destination node, start instance
20:35:53,880 node-1-2.db1 INFO start failover down instance to satisfy the placed@node-1-2 target
20:35:54,076 node-1-2.db1 INFO do start --local (daemon origin)
20:35:54,252 node-1-2.db1.ip#0 INFO checking 192.168.100.14 availability
20:35:59,323 node-1-2.db1.disk#0 INFO vgchange --deltag @node-1-1 db1
20:35:59,370 node-1-2.db1.disk#0 INFO Volume group "db1" successfully changed
20:35:59,370 node-1-2.db1.disk#0 INFO output:
20:35:59,372 node-1-2.db1.disk#0 INFO vgchange --addtag @node-1-2 db1
20:35:59,426 node-1-2.db1.disk#0 INFO output:
20:35:59,427 node-1-2.db1.disk#0 INFO Volume group "db1" successfully changed
20:35:59,539 node-1-2.db1.disk#0 INFO vgchange -a y db1
20:35:59,616 node-1-2.db1.disk#0 INFO 1 logical volume(s) in volume group "db1" now active
20:35:59,616 node-1-2.db1.disk#0 INFO output:
20:35:59,766 node-1-2.db1.disk#1 INFO lv db1/data is already up
20:35:59,861 node-1-2.db1.fs#1 INFO mount -t xfs /dev/db1/data /srv/db1/data
20:36:00,160 node-1-2.db1.container#0 INFO docker run --name=db1.container.0 --detach --net=none google/pause
20:36:00,436 node-1-2.db1.container#0 INFO output:
20:36:00,437 node-1-2.db1.container#0 INFO 8d4947414095d12186ea250b966cb8666d8734b5b8d481739ab1722e79927114
20:36:00,462 node-1-2.db1.container#0 INFO wait for up status
20:36:00,506 node-1-2.db1.container#0 INFO wait for container operational
20:36:00,671 node-1-2.db1.ip#0 INFO bridge mode
20:36:00,773 node-1-2.db1.ip#0 INFO /sbin/ip link add name veth0pl25524 mtu 1500 type veth peer name veth0pg25524 mtu 1500
20:36:00,798 node-1-2.db1.ip#0 INFO /sbin/ip link set veth0pl25524 master br-prd
20:36:00,812 node-1-2.db1.ip#0 INFO /sbin/ip link set veth0pl25524 up
20:36:00,824 node-1-2.db1.ip#0 INFO /sbin/ip link set veth0pg25524 netns 25524
20:36:00,835 node-1-2.db1.ip#0 INFO /usr/bin/nsenter --net=/var/run/docker/netns/2bbcf661b019 ip link set veth0pg25524 name eth0
20:36:00,843 node-1-2.db1.ip#0 INFO /usr/bin/nsenter --net=/var/run/docker/netns/2bbcf661b019 ip addr add 192.168.100.14/24 dev eth0
20:36:00,850 node-1-2.db1.ip#0 INFO /usr/bin/nsenter --net=/var/run/docker/netns/2bbcf661b019 ip link set eth0 up
20:36:00,867 node-1-2.db1.ip#0 INFO /usr/bin/nsenter --net=/var/run/docker/netns/2bbcf661b019 ip route replace default via 192.168.100.1
20:36:00,874 node-1-2.db1.ip#0 INFO /usr/bin/nsenter --net=/var/run/docker/netns/2bbcf661b019 /usr/bin/python /usr/share/opensvc/lib/arp.py eth0 192.168.100.14
20:36:01,267 node-1-2.db1.container#1 INFO docker run --name=db1.container.1 -v /srv/db1/data/postgresql:/var/lib/postgresql:rw --user=999:999 --detach --net=container:db1.container.0 postgres
20:36:01,480 node-1-2.db1.container#1 INFO output:
20:36:01,481 node-1-2.db1.container#1 INFO ee21ce46ffd922582228cb86d2cabc098e7d02141b87b65b210ec4c40f382d43
20:36:01,536 node-1-2.db1.container#1 INFO wait for up status
20:36:01,624 node-1-2.db1.container#1 INFO wait for container operational
Activate Automatic Failover¶
$ om db1 set --kw orchestrate=ha