Docker Multi Containers

Docker is a great tool, but you may find it painful if you work with complex applications that rely on multiple docker containers. You may have to deal with different docker image providers, who will deliver you their respective part of the application. You have to make all containers work together, and to be able to upgrade and test easily a new release of only one provider. You may also be responsible of scheduling the containers start and stop, meaning that you have to take care of containers dependencies during start and stop operations.

This tutorial explains how you can ease multiple container management using the OpenSVC infrastructure.

Pre-requisites

  • OpenSVC service managing docker containers (As described in the parent topic)
  • We assume that the docker images are available (It means that you already have docker pulled your needed images, check here if you need help for that part)

Let's take an example and consider that we are responsible for hosting an application relying on 3 docker containers :

  • busybox
  • shipyard/shipyard
  • dhrp/sshd

Service Configuration File

root@deb1:/# svcmgr -s app.opensvc.com print config
[DEFAULT]
autostart_node = deb1.opensvc.com
app = OSVCLAB
service_type = DEV
nodes = deb1.opensvc.com
docker_data_dir = /opt/app.opensvc.com/appdata
docker_daemon_args = --ip 37.59.71.25

[container#1]
type = docker
run_image = f66342b343ae
run_command = /bin/sh

[container#2]
type = docker
run_image = 30e0b59613ff
run_args = -v /var/lib/opensvc/app.opensvc.com/docker.sock:/docker.sock -p 8888:8000

[container#3]
type = docker
run_image = 2bbfe079a942
run_args = -v /opt/app.opensvc.com/vol1:/vol1:rw -p 42222:22

[ip#1]
ipdev = eth0
ipname = app.opensvc.com

[vg#1]
vgname = vgapp

[fs#1]
mnt_opt = rw
mnt = /opt/app.opensvc.com
dev = /dev/mapper/vgapp-lvapproot
type = ext4

[fs#2]
mnt_opt = rw
mnt = /opt/app.opensvc.com/appdata
dev = /dev/mapper/vgapp-lvappdata
type = ext4

We need to manage 3 containers to make this application working, so we declare 3 opensvc docker-type container resources:

  • [container#1] is a busybox container, and has to be launched first for our application. We just need to spawn /bin/sh in the container.
  • [container#2] is the shipyard container and is second in the startup sequencing. We have to present the docker socket inside the container (<OSVCVAR>/app.opensvc.com/docker.sock will be available in the container as /docker.sock), and expose network ports (internal container port 8000 will be available on host port 8888)
  • [container#3] is the dhrp/sshd and must be the last container to start. We present a data volume inside the container, and expose ssh port to host port 42222.

Service Startup

docker images are present in the local repository

root@deb1:/# svcmgr -s app.opensvc.com docker images | grep latest
busybox             latest              f66342b343ae        4 days ago          2.433 MB
shipyard/shipyard   latest              30e0b59613ff        5 weeks ago         516.5 MB
dhrp/sshd           latest              2bbfe079a942        11 months ago       321.8 MB

OpenSVC print status

root@deb1:/# svcmgr -s app.opensvc.com print status
app.opensvc.com
overall                   warn
|- avail                  warn
|  |- container#1    .... down     busybox:latest
|  |  |                            # can not find container id
|  |- container#2    .... down     shipyard/shipyard:latest
|  |  |                            # can not find container id
|  |- container#3    .... down     dhrp/sshd:latest
|  |  |                            # can not find container id
|  |- vg#1           .... up       vgapp
|  |- fs#1           .... up       /dev/mapper/vgapp-lvapproot@/opt/app.opensvc.com
|  |- fs#2           .... up       /dev/mapper/vgapp-lvappdata@/opt/app.opensvc.com/appdata
|  '- ip#1           .... up       app.opensvc.com@eth0
|- sync                   n/a
'- hb                     n/a

Note

As containers were never started, OpenSVC agent complains about missing container id.

run the containers

root@deb1:/# svcmgr -s app.opensvc.com start
11:56:19 INFO    APP.OPENSVC.COM.IP#1    checking 37.59.71.25 availability
11:56:19 INFO    APP.OPENSVC.COM.IP#1    37.59.71.25 is already up on eth0
11:56:19 INFO    APP.OPENSVC.COM.VG#1    vgapp is already up
11:56:19 INFO    APP.OPENSVC.COM.FS#1    fs(/dev/mapper/vgapp-lvapproot /opt/app.opensvc.com) is already mounted
11:56:19 INFO    APP.OPENSVC.COM.FS#2    fs(/dev/mapper/vgapp-lvappdata /opt/app.opensvc.com/appdata) is already mounted
11:56:19 INFO    APP.OPENSVC.COM.CONTAINER#1 docker -H unix:///var/lib/opensvc/app.opensvc.com/docker.sock run -t -i -d --name=app.opensvc.com.container.1 f66342b343ae /bin/sh
11:56:19 INFO    APP.OPENSVC.COM.CONTAINER#1 output:
2d4b80bdec715a7df33bc0bb36b8e65598e41d83d0b8919b717287b3ca9c9dae

11:56:19 INFO    APP.OPENSVC.COM.CONTAINER#1 wait for container up status
11:56:19 INFO    APP.OPENSVC.COM.CONTAINER#1 wait for container operational
11:56:19 INFO    APP.OPENSVC.COM.CONTAINER#2 docker -H unix:///var/lib/opensvc/app.opensvc.com/docker.sock run -t -i -d --name=app.opensvc.com.container.2 -v /var/lib/opensvc/app.opensvc.com/docker.sock:/docker.sock -p 8888:8000 30e0b59613ff
11:56:19 INFO    APP.OPENSVC.COM.CONTAINER#2 output:
cecf54c6c76977d7d8b5801803d270023023efcc9690860da56d6560a81800e6

11:56:19 INFO    APP.OPENSVC.COM.CONTAINER#2 wait for container up status
11:56:19 INFO    APP.OPENSVC.COM.CONTAINER#2 wait for container operational
11:56:19 INFO    APP.OPENSVC.COM.CONTAINER#3 docker -H unix:///var/lib/opensvc/app.opensvc.com/docker.sock run -t -i -d --name=app.opensvc.com.container.3 -v /opt/app.opensvc.com/vol1:/vol1:rw -p 42222:22 2bbfe079a942
11:56:19 INFO    APP.OPENSVC.COM.CONTAINER#3 output:
c0a390acb80a2e42bb57da5a104dc23ec34955e2aec4ac3cf270399447f64c5f

11:56:19 INFO    APP.OPENSVC.COM.CONTAINER#3 wait for container up status
11:56:19 INFO    APP.OPENSVC.COM.CONTAINER#3 wait for container operational

root@deb1:/# svcmgr -s app.opensvc.com print status
app.opensvc.com
overall                   up
|- avail                  up
|  |- container#1    .... up       2d4b80bdec71@busybox:latest
|  |- container#2    .... up       cecf54c6c769@shipyard/shipyard:latest
|  |- container#3    .... up       c0a390acb80a@dhrp/sshd:latest
|  |- vg#1           .... up       vgapp
|  |- fs#1           .... up       /dev/mapper/vgapp-lvapproot@/opt/app.opensvc.com
|  |- fs#2           .... up       /dev/mapper/vgapp-lvappdata@/opt/app.opensvc.com/appdata
|  '- ip#1           .... up       app.opensvc.com@eth0
|- sync                   n/a
'- hb                     n/a

Note

OpenSVC agent start containers in sequence. Please note timestamps : docker's efficiency associated with OpenSVC allow the start/stop of complex environments in a few seconds.

Querying docker daemon about containers

root@deb1:/# svcmgr -s app.opensvc.com docker ps -a
CONTAINER ID        IMAGE                      COMMAND               CREATED             STATUS              PORTS                        NAMES
c0a390acb80a        dhrp/sshd:latest           /usr/sbin/sshd -D     34 seconds ago      Up 33 seconds       37.59.71.25:42222->22/tcp    app.opensvc.com.container.3
cecf54c6c769        shipyard/shipyard:latest   /app/.docker/run.sh   34 seconds ago      Up 33 seconds       37.59.71.25:8888->8000/tcp   app.opensvc.com.container.2
2d4b80bdec71        busybox:latest             /bin/sh               34 seconds ago      Up 33 seconds                                    app.opensvc.com.container.1

Note

We see that our containers are running as expected, and networks ports are bound to the OpenSVC service IP address

Containers Management

All containers

It is possible to manage containers together, either start or stop, by using the startcontainer or stopcontainer OpenSVC keywords.

root@deb1:/# svcmgr -s app.opensvc.com print status
app.opensvc.com
overall                   warn
|- avail                  warn
|  |- container#1    .... down     f66342b343ae
|  |  |                            # docker daemon is not running
|  |- container#2    .... down     30e0b59613ff
|  |  |                            # docker daemon is not running
|  |- container#3    .... down     2bbfe079a942
|  |  |                            # docker daemon is not running
|  |- vg#1           .... up       vgapp
|  |- fs#1           .... up       /dev/mapper/vgapp-lvapproot@/opt/app.opensvc.com
|  |- fs#2           .... up       /dev/mapper/vgapp-lvappdata@/opt/app.opensvc.com/appdata
|  '- ip#1           .... up       app.opensvc.com@eth0
|- sync                   n/a
'- hb                     n/a

root@deb1:/# svcmgr -s app.opensvc.com docker ps -a
CONTAINER ID        IMAGE                      COMMAND               CREATED             STATUS                       PORTS                        NAMES
c0a390acb80a        dhrp/sshd:latest           /usr/sbin/sshd -D     2 hours ago         Exited (0) 38 seconds ago    37.59.71.25:42222->22/tcp    app.opensvc.com.container.3
cecf54c6c769        shipyard/shipyard:latest   /app/.docker/run.sh   2 hours ago         Exited (0) 28 seconds ago   37.59.71.25:8888->8000/tcp   app.opensvc.com.container.2
2d4b80bdec71        busybox:latest             /bin/sh               2 hours ago         Exited (0) 18 seconds ago                                app.opensvc.com.container.1

root@deb1:/# svcmgr -s app.opensvc.com startcontainer
13:56:29 INFO    APP.OPENSVC.COM.CONTAINER#1 docker -H unix:///var/lib/opensvc/app.opensvc.com/docker.sock start 2d4b80bdec71
13:56:29 INFO    APP.OPENSVC.COM.CONTAINER#1 output:
2d4b80bdec71

13:56:29 INFO    APP.OPENSVC.COM.CONTAINER#1 wait for container up status
13:56:29 INFO    APP.OPENSVC.COM.CONTAINER#1 wait for container operational
13:56:29 INFO    APP.OPENSVC.COM.CONTAINER#2 docker -H unix:///var/lib/opensvc/app.opensvc.com/docker.sock start cecf54c6c769
13:56:29 INFO    APP.OPENSVC.COM.CONTAINER#2 output:
cecf54c6c769

13:56:29 INFO    APP.OPENSVC.COM.CONTAINER#2 wait for container up status
13:56:29 INFO    APP.OPENSVC.COM.CONTAINER#2 wait for container operational
13:56:29 INFO    APP.OPENSVC.COM.CONTAINER#3 docker -H unix:///var/lib/opensvc/app.opensvc.com/docker.sock start c0a390acb80a
13:56:30 INFO    APP.OPENSVC.COM.CONTAINER#3 output:
c0a390acb80a

13:56:30 INFO    APP.OPENSVC.COM.CONTAINER#3 wait for container up status
13:56:30 INFO    APP.OPENSVC.COM.CONTAINER#3 wait for container operational

root@deb1:/# svcmgr -s app.opensvc.com print status
app.opensvc.com
overall                   up
|- avail                  up
|  |- container#1    .... up       2d4b80bdec71@busybox:latest
|  |- container#2    .... up       cecf54c6c769@shipyard/shipyard:latest
|  |- container#3    .... up       c0a390acb80a@dhrp/sshd:latest
|  |- vg#1           .... up       vgapp
|  |- fs#1           .... up       /dev/mapper/vgapp-lvapproot@/opt/app.opensvc.com
|  |- fs#2           .... up       /dev/mapper/vgapp-lvappdata@/opt/app.opensvc.com/appdata
|  '- ip#1           .... up       app.opensvc.com@eth0
|- sync                   n/a
'- hb                     n/a

root@deb1:/# svcmgr -s app.opensvc.com docker ps -a
CONTAINER ID        IMAGE                      COMMAND               CREATED             STATUS              PORTS                        NAMES
c0a390acb80a        dhrp/sshd:latest           /usr/sbin/sshd -D     2 hours ago         Up 12 seconds       37.59.71.25:42222->22/tcp    app.opensvc.com.container.3
cecf54c6c769        shipyard/shipyard:latest   /app/.docker/run.sh   2 hours ago         Up 12 seconds       37.59.71.25:8888->8000/tcp   app.opensvc.com.container.2
2d4b80bdec71        busybox:latest             /bin/sh               2 hours ago         Up 12 seconds                                    app.opensvc.com.container.1

One container

Like any other OpenSVC resource in the service configuration file, it is possible to manage it with its resource ID, by using the --rid <resource_id> option.

root@deb1:/# svcmgr -s app.opensvc.com print status
app.opensvc.com
overall                   up
|- avail                  up
|  |- container#1    .... up       2d4b80bdec71@busybox:latest
|  |- container#2    .... up       cecf54c6c769@shipyard/shipyard:latest
|  |- container#3    .... up       c0a390acb80a@dhrp/sshd:latest
|  |- vg#1           .... up       vgapp
|  |- fs#1           .... up       /dev/mapper/vgapp-lvapproot@/opt/app.opensvc.com
|  |- fs#2           .... up       /dev/mapper/vgapp-lvappdata@/opt/app.opensvc.com/appdata
|  '- ip#1           .... up       app.opensvc.com@eth0
|- sync                   n/a
'- hb                     n/a

root@deb1:/# svcmgr -s app.opensvc.com stop --rid container#1
14:07:05 INFO    APP.OPENSVC.COM.CONTAINER#1 docker -H unix:///var/lib/opensvc/app.opensvc.com/docker.sock stop 2d4b80bdec71
14:07:15 INFO    APP.OPENSVC.COM.CONTAINER#1 output:
2d4b80bdec71

14:07:15 INFO    APP.OPENSVC.COM.CONTAINER#1 wait for container down status

root@deb1:/# svcmgr -s app.opensvc.com print status
app.opensvc.com
overall                   warn
|- avail                  warn
|  |- container#1    .... down     2d4b80bdec71@busybox:latest
|  |- container#2    .... up       cecf54c6c769@shipyard/shipyard:latest
|  |- container#3    .... up       c0a390acb80a@dhrp/sshd:latest
|  |- vg#1           .... up       vgapp
|  |- fs#1           .... up       /dev/mapper/vgapp-lvapproot@/opt/app.opensvc.com
|  |- fs#2           .... up       /dev/mapper/vgapp-lvappdata@/opt/app.opensvc.com/appdata
|  '- ip#1           .... up       app.opensvc.com@eth0
|- sync                   n/a
'- hb                     n/a

root@deb1:/# svcmgr -s app.opensvc.com docker ps -a
CONTAINER ID        IMAGE                      COMMAND               CREATED             STATUS                       PORTS                        NAMES
c0a390acb80a        dhrp/sshd:latest           /usr/sbin/sshd -D     2 hours ago         Up 10 minutes                37.59.71.25:42222->22/tcp    app.opensvc.com.container.3
cecf54c6c769        shipyard/shipyard:latest   /app/.docker/run.sh   2 hours ago         Up 10 minutes                37.59.71.25:8888->8000/tcp   app.opensvc.com.container.2
2d4b80bdec71        busybox:latest             /bin/sh               2 hours ago         Exited (-1) 12 seconds ago                                app.opensvc.com.container.1

root@deb1:/# svcmgr -s app.opensvc.com start --rid container#1
14:07:45 INFO    APP.OPENSVC.COM.IP#1    checking 37.59.71.25 availability
14:07:45 INFO    APP.OPENSVC.COM.CONTAINER#1 docker -H unix:///var/lib/opensvc/app.opensvc.com/docker.sock start 2d4b80bdec71
14:07:45 INFO    APP.OPENSVC.COM.CONTAINER#1 output:
2d4b80bdec71

14:07:45 INFO    APP.OPENSVC.COM.CONTAINER#1 wait for container up status
14:07:45 INFO    APP.OPENSVC.COM.CONTAINER#1 wait for container operational

root@deb1:/# svcmgr -s app.opensvc.com print status
app.opensvc.com
overall                   up
|- avail                  up
|  |- container#1    .... up       2d4b80bdec71@busybox:latest
|  |- container#2    .... up       cecf54c6c769@shipyard/shipyard:latest
|  |- container#3    .... up       c0a390acb80a@dhrp/sshd:latest
|  |- vg#1           .... up       vgapp
|  |- fs#1           .... up       /dev/mapper/vgapp-lvapproot@/opt/app.opensvc.com
|  |- fs#2           .... up       /dev/mapper/vgapp-lvappdata@/opt/app.opensvc.com/appdata
|  '- ip#1           .... up       app.opensvc.com@eth0
|- sync                   n/a
'- hb                     n/a

Container Upgrade

Let's consider that you have to change from official image busybox/latest to google/busybox, which simulate the fact that someone provide you a new version of docker image to run instead of the old one, the steps would be :

Stop the container to upgrade:

root@deb1:/# svcmgr -s app.opensvc.com stop --rid container#1
14:27:35 INFO    APP.OPENSVC.COM.CONTAINER#1 docker -H unix:///var/lib/opensvc/app.opensvc.com/docker.sock stop 2d4b80bdec71
14:27:45 INFO    APP.OPENSVC.COM.CONTAINER#1 output:
2d4b80bdec71

14:27:45 INFO    APP.OPENSVC.COM.CONTAINER#1 wait for container down status

Pull the new docker image:

root@deb1:/# svcmgr -s app.opensvc.com docker pull google/busybox
Pulling repository google/busybox
c8f0cfead624: Download complete
511136ea3c5a: Download complete
bd5d7a592a52: Download complete
31ea9d8cd4b6: Download complete

root@deb1:/# svcmgr -s app.opensvc.com docker images | grep google
google/busybox      latest              c8f0cfead624        2 weeks ago         2.491 MB

Note

The image is in the local repository, with image id c8f0cfead624. We need this image id to modify OpenSVC service configuration file.

Remove the old container:

root@deb1:/# svcmgr -s app.opensvc.com docker ps -a
CONTAINER ID        IMAGE                      COMMAND               CREATED             STATUS                       PORTS                        NAMES
c0a390acb80a        dhrp/sshd:latest           /usr/sbin/sshd -D     2 hours ago         Up 32 minutes                37.59.71.25:42222->22/tcp    app.opensvc.com.container.3
cecf54c6c769        shipyard/shipyard:latest   /app/.docker/run.sh   2 hours ago         Up 32 minutes                37.59.71.25:8888->8000/tcp   app.opensvc.com.container.2
2d4b80bdec71        busybox:latest             /bin/sh               2 hours ago         Exited (-1) 58 seconds ago                                app.opensvc.com.container.1

root@deb1:/# svcmgr -s app.opensvc.com docker rm 2d4b80bdec71
2d4b80bdec71

Note

It could be a good idea to rename our container instead of delete it, for rollback purposes. At the beginning of June 2014, Docker doesn't support that, but it's on the roadmap.

Modify OpenSVC configuration:

root@deb1:/# cd /etc/opensvc && cp app.opensvc.com.env app.opensvc.com.cfgsave

root@deb1:/# cat app.opensvc.com.cfgsave | sed -e 's/f66342b343ae/c8f0cfead624/' > app.opensvc.com.env

root@deb1:/# grep -C2 c8f0cfead624 app.opensvc.com.env
[container#1]
type = docker
run_image = c8f0cfead624
run_command = /bin/sh

Note

We just have to change the run_image parameter to point to the new image to execute in the container

Bring back service with new container:

root@deb1:/# svcmgr -s app.opensvc.com start --rid container#1
send /etc/opensvc/app.opensvc.com.env to collector ... OK
update /var/lib/opensvc/app.opensvc.com.push timestamp ... OK
14:32:25 INFO    APP.OPENSVC.COM.IP#1    checking 37.59.71.25 availability
14:32:25 INFO    APP.OPENSVC.COM.CONTAINER#1 docker -H unix:///var/lib/opensvc/app.opensvc.com/docker.sock run -t -i -d --name=app.opensvc.com.container.1 c8f0cfead624 /bin/sh
14:32:25 INFO    APP.OPENSVC.COM.CONTAINER#1 output:
dc32fbc0a6c7e1e4f981fef6444b50bb9b3add5103062b91a15716cd5396ee43

14:32:25 INFO    APP.OPENSVC.COM.CONTAINER#1 wait for container up status
14:32:25 INFO    APP.OPENSVC.COM.CONTAINER#1 wait for container operational

root@deb1:/# svcmgr -s app.opensvc.com print status
app.opensvc.com
overall                   up
|- avail                  up
|  |- container#1    .... up       dc32fbc0a6c7@google/busybox:latest
|  |- container#2    .... up       cecf54c6c769@shipyard/shipyard:latest
|  |- container#3    .... up       c0a390acb80a@dhrp/sshd:latest
|  |- vg#1           .... up       vgapp
|  |- fs#1           .... up       /dev/mapper/vgapp-lvapproot@/opt/app.opensvc.com
|  |- fs#2           .... up       /dev/mapper/vgapp-lvappdata@/opt/app.opensvc.com/appdata
|  '- ip#1           .... up       app.opensvc.com@eth0
|- sync                   n/a
'- hb                     n/a

Complex Orchestration

When implementing applications as Docker containers, you may be constrained to organize container startup in a specific order, because it's the way your application works. Another kind of problem is that you may need to start multiple containers in parrallel instead of sequentially. OpenSVC agent is developped to support those specific constraints.

The following example is a dummy application made of :

  • 1 database server : this container have to be started first, and stopped at the very end.
  • 4 application servers : those containers have to be started in parrallel, and just after the database server.
  • 2 webservers : last to be launched, just after the appservers.

Considering that it is an example, all containers will depend on image id b073e328878e, which corresponds to docker public image opensvc/busybox:date

Note

if one of the container meet start failure issue, the overall service rollback and stop

OpenSVC Syntax

The method to implement such a modelization rely on 2 configuration tricks:

  • tag container resource in container subsets:

    subset = 01appservers
    
  • optionaly declare a parallel start on a per subset basis:

    [subset#container.docker:01appservers]
    parallel = true
    

Subsets are processed in alphanumerical order. We recommend prefixing the subset name with a 2-digits number, to improve configuration readability.

Example

Let's have a look at our dummy application service configuration file

[DEFAULT]
autostart_node = deb1.opensvc.com
app = OSVCLAB
service_type = DEV
nodes = deb1.opensvc.com
docker_data_dir = /opt/busybox.opensvc.com/appdata
docker_daemon_args = --ip 37.59.71.25

[container#1]
type = docker
run_image = b073e328878e
subset = 00database

[container#2]
type = docker
run_image = b073e328878e
subset = 01appservers

[container#3]
type = docker
run_image = b073e328878e
subset = 01appservers

[container#4]
type = docker
run_image = b073e328878e
subset = 01appservers

[container#5]
type = docker
run_image = b073e328878e
subset = 01appservers

[subset#container.docker:01appservers]
parallel = true

[container#6]
type = docker
run_image = b073e328878e
subset = 02webservers

[container#7]
type = docker
run_image = b073e328878e
subset = 02webservers

[subset#container.docker:02webservers]
parallel = false

[ip#1]
ipdev = eth0
ipname = busybox.opensvc.com

[vg#1]
vgname = vgbusybox
scsireserv = false

[fs#1]
mnt_opt = rw
mnt = /opt/busybox.opensvc.com
dev = /dev/mapper/vgbusybox-lvbusyboxroot
type = ext4

[fs#2]
mnt_opt = rw
mnt = /opt/busybox.opensvc.com/appdata
dev = /dev/mapper/vgbusybox-lvbusyboxdata
type = ext4

Service startup

Actions are processed in the following order

  • ip/vg/fs start
  • subset 00database start
  • subset 01appservers parallel start
  • subset 02webservers sequential start
root@deb1:/# busybox.opensvc.com print status
busybox.opensvc.com
overall                   down
|- avail                  down
|  |- container#1    .... down     b073e328878e
|  |  |                            # docker daemon is not running
|  |- container#2    .... down     b073e328878e
|  |  |                            # docker daemon is not running
|  |- container#3    .... down     b073e328878e
|  |  |                            # docker daemon is not running
|  |- container#4    .... down     b073e328878e
|  |  |                            # docker daemon is not running
|  |- container#5    .... down     b073e328878e
|  |  |                            # docker daemon is not running
|  |- container#6    .... down     b073e328878e
|  |  |                            # docker daemon is not running
|  |- container#7    .... down     b073e328878e
|  |  |                            # docker daemon is not running
|  |- vg#1           .... down     vgbusybox
|  |- fs#1           .... down     /dev/mapper/vgbusybox-lvbusyboxroot@/opt/busybox.opensvc.com
|  |- fs#2           .... down     /dev/mapper/vgbusybox-lvbusyboxdata@/opt/busybox.opensvc.com/appdata
|  '- ip#1           .... down     busybox.opensvc.com@eth0
|- sync                   n/a
'- hb                     n/a

root@deb1:/# busybox.opensvc.com start
09:10:18 INFO    BUSYBOX.OPENSVC.COM.IP#1    checking 37.59.71.25 availability
09:10:21 INFO    BUSYBOX.OPENSVC.COM.IP#1    ifconfig eth0:1 37.59.71.25 netmask 255.255.255.224 up
09:10:21 INFO    BUSYBOX.OPENSVC.COM.IP#1    arping -U -c 1 -I eth0 -s 37.59.71.25 37.59.71.25
09:10:21 INFO    BUSYBOX.OPENSVC.COM.VG#1    vgchange --addtag @deb1.opensvc.com vgbusybox
09:10:22 INFO    BUSYBOX.OPENSVC.COM.VG#1    output:
  Volume group "vgbusybox" successfully changed

09:10:22 INFO    BUSYBOX.OPENSVC.COM.VG#1    vgchange -a y vgbusybox
09:10:22 INFO    BUSYBOX.OPENSVC.COM.VG#1    output:
  2 logical volume(s) in volume group "vgbusybox" now active

09:10:22 INFO    BUSYBOX.OPENSVC.COM.FS#1    e2fsck -p /dev/mapper/vgbusybox-lvbusyboxroot
09:10:22 INFO    BUSYBOX.OPENSVC.COM.FS#1    output:
/dev/mapper/vgbusybox-lvbusyboxroot: clean, 13/65536 files, 12637/262144 blocks

09:10:22 INFO    BUSYBOX.OPENSVC.COM.FS#1    mount -t ext4 -o rw /dev/mapper/vgbusybox-lvbusyboxroot /opt/busybox.opensvc.com
09:10:22 INFO    BUSYBOX.OPENSVC.COM.FS#2    e2fsck -p /dev/mapper/vgbusybox-lvbusyboxdata
09:10:22 INFO    BUSYBOX.OPENSVC.COM.FS#2    output:
/dev/mapper/vgbusybox-lvbusyboxdata: clean, 656/65536 files, 25170/262144 blocks

09:10:22 INFO    BUSYBOX.OPENSVC.COM.FS#2    mount -t ext4 /dev/mapper/vgbusybox-lvbusyboxdata /opt/busybox.opensvc.com/appdata
09:10:22 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:00DATABASE#1 starting docker daemon
09:10:22 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:00DATABASE#1 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock -r=false -d -g /opt/busybox.opensvc.com/appdata -p /var/lib/opensvc/busybox.opensvc.com/docker.pid --ip 37.59.71.25
09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:00DATABASE#1 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock start b82cf3232b79
09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:00DATABASE#1 output:
b82cf3232b79

09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:00DATABASE#1 wait for container up status
09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:00DATABASE#1 wait for container operational
09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#2 action start started in child process 23635
09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#3 action start started in child process 23636
09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#4 action start started in child process 23638
09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#5 action start started in child process 23640
09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#3 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock start 185751ce205b
09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#4 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock start 6212757a24c6
09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#5 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock start 68b2e591147a
09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#2 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock start 7e0f85484429
09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#3 output:
185751ce205b

09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#4 output:
6212757a24c6

09:10:23 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#3 wait for container up status
09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#4 wait for container up status
09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#3 wait for container operational
09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#5 output:
68b2e591147a

09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#5 wait for container up status
09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#4 wait for container operational
09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#2 output:
7e0f85484429

09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#2 wait for container up status
09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#5 wait for container operational
09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#2 wait for container operational
09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:02WEBSERVERS#6 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock start 6b82e882acf0
09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:02WEBSERVERS#6 output:
6b82e882acf0

09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:02WEBSERVERS#6 wait for container up status
09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:02WEBSERVERS#6 wait for container operational
09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:02WEBSERVERS#7 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock start a825bb126088
09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:02WEBSERVERS#7 output:
a825bb126088

09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:02WEBSERVERS#7 wait for container up status
09:10:24 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:02WEBSERVERS#7 wait for container operational

root@deb1:/# busybox.opensvc.com print status
busybox.opensvc.com
overall                   up
|- avail                  up
|  |- container#1    .... up       b82cf3232b79@opensvc/busybox:date
|  |- container#2    .... up       7e0f85484429@opensvc/busybox:date
|  |- container#3    .... up       185751ce205b@opensvc/busybox:date
|  |- container#4    .... up       6212757a24c6@opensvc/busybox:date
|  |- container#5    .... up       68b2e591147a@opensvc/busybox:date
|  |- container#6    .... up       6b82e882acf0@opensvc/busybox:date
|  |- container#7    .... up       a825bb126088@opensvc/busybox:date
|  |- vg#1           .... up       vgbusybox
|  |- fs#1           .... up       /dev/mapper/vgbusybox-lvbusyboxroot@/opt/busybox.opensvc.com
|  |- fs#2           .... up       /dev/mapper/vgbusybox-lvbusyboxdata@/opt/busybox.opensvc.com/appdata
|  '- ip#1           .... up       busybox.opensvc.com@eth0
|- sync                   n/a
'- hb                     n/a

Service stop

Actions are processed in the following order

  • subset 02webservers sequential stop
  • subset 01appservers parallel stop
  • subset 00database stop
  • ip/vg/fs stop
root@deb1:/# busybox.opensvc.com stop
09:10:40 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:02WEBSERVERS#7 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock stop a825bb126088
09:10:50 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:02WEBSERVERS#7 output:
a825bb126088

09:10:50 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:02WEBSERVERS#7 wait for container down status
09:10:51 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:02WEBSERVERS#6 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock stop 6b82e882acf0
09:11:01 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:02WEBSERVERS#6 output:
6b82e882acf0

09:11:01 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:02WEBSERVERS#6 wait for container down status
09:11:01 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#5 action stop started in child process 27048
09:11:01 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#4 action stop started in child process 27049
09:11:01 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#3 action stop started in child process 27051
09:11:01 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#2 action stop started in child process 27052
09:11:01 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#3 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock stop 185751ce205b
09:11:01 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#4 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock stop 6212757a24c6
09:11:01 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#2 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock stop 7e0f85484429
09:11:01 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#5 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock stop 68b2e591147a
09:11:11 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#2 output:
7e0f85484429

09:11:11 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#2 wait for container down status
09:11:11 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#3 output:
185751ce205b

09:11:11 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#3 wait for container down status
09:11:11 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#5 output:
68b2e591147a

09:11:11 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#5 wait for container down status
09:11:11 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#4 output:
6212757a24c6

09:11:11 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:01APPSERVERS#4 wait for container down status
09:11:11 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:00DATABASE#1 docker -H unix:///var/lib/opensvc/busybox.opensvc.com/docker.sock stop b82cf3232b79
09:11:21 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:00DATABASE#1 output:
b82cf3232b79

09:11:21 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:00DATABASE#1 wait for container down status
09:11:21 INFO    BUSYBOX.OPENSVC.COM.CONTAINER.DOCKER:00DATABASE#1 no more container handled by docker daemon. shut it down
09:11:21 INFO    BUSYBOX.OPENSVC.COM.FS#2    umount /opt/busybox.opensvc.com/appdata
09:11:22 INFO    BUSYBOX.OPENSVC.COM.FS#1    umount /opt/busybox.opensvc.com
09:11:22 INFO    BUSYBOX.OPENSVC.COM.VG#1    vgchange --deltag @deb1.opensvc.com vgbusybox
09:11:23 INFO    BUSYBOX.OPENSVC.COM.VG#1    output:
  Volume group "vgbusybox" successfully changed

09:11:23 INFO    BUSYBOX.OPENSVC.COM.VG#1    kpartx -d /dev/vgbusybox/lvbusyboxdata
09:11:23 INFO    BUSYBOX.OPENSVC.COM.VG#1    kpartx -d /dev/vgbusybox/lvbusyboxroot
09:11:23 INFO    BUSYBOX.OPENSVC.COM.VG#1    vgchange -a n vgbusybox
09:11:23 INFO    BUSYBOX.OPENSVC.COM.VG#1    output:
  0 logical volume(s) in volume group "vgbusybox" now active

09:11:23 INFO    BUSYBOX.OPENSVC.COM.IP#1    ifconfig eth0:1 down
09:11:23 INFO    BUSYBOX.OPENSVC.COM.IP#1    checking 37.59.71.25 availability