Provisioning¶
Introduction¶
Services and volumes objecs handle resources activation and desactivation: disk, volumes groups, filesystems for example. Upon creation, the resources may or may not already exist. In the later case, the object can provision them, in other words the object can allocate disks and ip address, create volume groups and logical volume, format filesystems, create containers, pull docker images.
This section describes:
how this provisioning feature work
how to provision and template services
some useful examples
Provisioners¶
Each resource driver can implement a provisioner, but not all resources actually do. The list of drivers with provisioning keywords:
$ egrep -l "provisioning:\s*True" <OSVCDOC>/template.*conf | sed -e "s@<OSVCDOC>/template.service.@@" -e "s@.conf@@" | grep -v "fs\."
container.esx
container.kvm
container.lxc
container.lxd
container.ovm
container.srp
container.vz
container.xen
container.zone
disk.disk
disk.gce
disk.loop
disk.lv
disk.lvm
disk.md
disk.pool
disk.rados
disk.veritas
disk.vg
disk.vxdg
disk.vxvol
disk.zpool
disk.zvol
fs
ip.amazon
ip
ip.crossbow
ip.docker
ip.gce
ip.netns
volume
Each resource provisioner may require or support additional parameters. The resource configuration templates in <OSVCDOC>
highlight these particular parameters with provisioning = True
. The list of provisioning keywords of the fs.xfs
driver can thus be extracted using:
$ egrep -B 5 "provisioning:\s*True" <OSVCDOC>/template.fs.xfs.conf \
| grep keyword
# keyword: vg
# keyword: size
A provisioner can update the DEFAULT and resource configuration parameters. For example, most ip provisioners set the resource ipname
parameter.
Provisioning Workflow¶
The provisioning can be either local (–local) or orchestrated.
When orchestrated, the daemon starts provisioning the instance on the placement leaders. When provisioned, these instances stays up
and the daemons move on to provisioning all other instances in parallel. Those are rollbacked after provisioned, so the service is in optimal state at the end of the orchestration.
The CRM command run on the leader node is:
$ om <path> provision --local --disable-rollback --leader
The CRM command run on the non-leader nodes is:
$ om <path> provision --local
The provisioners are run in the start natural order. Each resource is left in the up
state after its provisioning, so that the following provisioners can count on their availability to proceed. For example, a amazon allocated disk must be left attached for a fs provisioner to format it.
Provisioning Usage¶
The provisioners are activated either by the provision
and deploy
action or by setting the --provision
option with the following actions.
Create¶
om <path> create --template <uri>|<template>
Creates a service using a configuration file pointed by
--template
.<uri>
being a local or remote path.<template>
being a collector served template id or template name. Served templates can be searched withom node collector search --like prov:<substring>
om <path> create --config <uri>
Creates a service using a configuration file pointed by
--config <uri>
.<uri>
being a local or remote path.om <path> create --resource <json definition> ...
Creates a service using definitions passed as
--resource
arguments.
Each --resource
argument takes a JSON formatted dictionary. The keys are the resource or DEFAULT section normal parameters, plus extra keys described in the following table.
Key |
Description |
---|---|
rtype |
The resource main type. The resource identifier (rid) is formed of this rtype concatenated with |
The create
service action will take care of the etc/mysvc*
directories and symlinks creation and the provisioning if --provision
is set.
Example:
$ om mysvc create --config /etc/opensvc/mysvc.conf --provision
Update¶
Add or change a resource definition to an existing service. Definitions are passed as --resource
arguments.
Pull¶
Creates a service using the configuration file of the service fetched from the collector.
Service Templates¶
A template is a normal service configuration file with parts you can replace with references and/or arithmetic evaluations. Templates can be stored in the local fs, served through ftp, http, https, or served by the collector with publications ACL.
A template is instanciated by copying its content as a service configuration file (<OSVCETC>/<svcname>.conf
or <OSVCETC>/<namespace>/<kind>/<name>.conf
).
Arithmetic Expressions¶
The arithmetic evaluation format is $(<expr>)
. An evaluation can contain references.
References¶
The reference format is {[<section>.]<option>}
, where section is a configuration file section name, and <option> is the option name in the pointed section. If section is ommited, the DEFAULT
section is implicitely used. A reference can also contain arithmetic evaluations.
Voir aussi
Env section¶
References to the env
section are special:
Options in the
env
are not submitted the synthaxic checks run byom <path> validate config
andom <path> edit
.om <path> create --interactive
prompts for each env key value, suggesting the value set in the template as default.om <path> create --env <option>=<value>
overrides the env options default values.System’s uppercased environment variables override the env options default values and values specified with
--env
.
Provisioning Examples¶
Docker Service on Amazon¶
Template testec2docker.template
:
[DEFAULT]
env = TST
nodes = node12.nsx.lab.net
docker_data_dir = /srv/{svcname}/docker
[ip#0]
ipname = <allocate>
#eip = <allocate>
type = amazon
ipdev = eth0
docker_daemon_ip = true
cascade_allocation = ip#1.ipname
[ip#1]
ipname = localhost
ipdev = eth0
[disk#0]
type = amazon
volumes = <size=15>
[fs#0]
type = btrfs
mnt_opt = defaults,subvol=docker
mnt = /srv/{svcname}/docker
standby = true
dev = /var/lib/opensvc/{svcname}/dev/disk.0.0
[fs#1]
type = btrfs
mnt_opt = defaults,subvol=data
mnt = /srv/{svcname}/data
dev = /var/lib/opensvc/{svcname}/dev/disk.0.0
[container#0]
image = ubuntu:14.10
type = docker
run_args = --net=bridge -p 80:80
command = /bin/bash
[container#1]
image = nginx:latest
type = docker
netns = container#0
[sync#0]
type = docker
target = nodes drpnodes
[sync#1]
src = {svcname}.fs.0:data
dst = {svcname}.fs.0:data
type = btrfs
target = drpnodes
Provision:
$ om testec2docker4.nsx.lab.net --config testec2docker.template --provision create --leader --disable-rollback
INFO testec2docker4.nsx.lab.net om testec2docker4.nsx.lab.net --config /etc/opensvc/testec2docker4.nsx.lab.net.conf --provision create
INFO testec2docker4.nsx.lab.net.ip#0 aws --output=json ec2 assign-private-ip-addresses --network-interface-id eni-033adc4b --secondary-private-ip-address-count 1
INFO testec2docker4.nsx.lab.net.ip#0 public ip already provisioned
INFO testec2docker4.nsx.lab.net.ip#0 cascade 10.0.0.221 to ip#1.ipname
INFO testec2docker4.nsx.lab.net.ip#0 provisioned
INFO testec2docker4.nsx.lab.net.ip#0 ec2 ip 10.0.0.221 is already assigned to this node
INFO testec2docker4.nsx.lab.net.ip#1 checking 10.0.0.221 availability
INFO testec2docker4.nsx.lab.net.ip#1 ifconfig eth0:3 10.0.0.221 netmask 255.255.255.0 up
INFO testec2docker4.nsx.lab.net.ip#1 arping -U -c 1 -I eth0 -s 10.0.0.221 10.0.0.221
INFO testec2docker4.nsx.lab.net.disk#0 aws --output=json ec2 create-volume --size 15 --availability-zone us-west-2b
INFO testec2docker4.nsx.lab.net.disk#0 vol-464a8f87 state: creating
INFO testec2docker4.nsx.lab.net.disk#0 vol-464a8f87 state: creating
INFO testec2docker4.nsx.lab.net.disk#0 vol-464a8f87 state: available
INFO testec2docker4.nsx.lab.net.disk#0 provisioned
INFO testec2docker4.nsx.lab.net.disk#0 aws --output=json ec2 attach-volume --instance-id i-40753486 --volume-id vol-464a8f87 --device /dev/sdh
INFO testec2docker4.nsx.lab.net.disk#0 /dev/xvdh is not present yet
INFO testec2docker4.nsx.lab.net.disk#0 /dev/xvdh is not present yet
INFO testec2docker4.nsx.lab.net.disk#0 /dev/xvdh is not present yet
INFO testec2docker4.nsx.lab.net.disk#0 /dev/xvdh is not present yet
INFO testec2docker4.nsx.lab.net.disk#0 /dev/xvdh is not present yet
INFO testec2docker4.nsx.lab.net.fs#1 /var/lib/opensvc/testec2docker4.nsx.lab.net/dev/disk.0.0 is not formatted
INFO testec2docker4.nsx.lab.net.fs#1 mkfs.btrfs /var/lib/opensvc/testec2docker4.nsx.lab.net/dev/disk.0.0
WARNING testec2docker4.nsx.lab.net.fs#1 command succesful but stderr:
Turning ON incompat feature 'extref': increased hardlink limit per file to 65536
INFO testec2docker4.nsx.lab.net.fs#1 output:
WARNING! - Btrfs v3.12 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /var/lib/opensvc/testec2docker4.nsx.lab.net/dev/disk.0.0
nodesize 16384 leafsize 16384 sectorsize 4096 size 15.00GiB
Btrfs v3.12
INFO testec2docker4.nsx.lab.net.fs#1 mount -t btrfs -o subvolid=0 /var/lib/opensvc/testec2docker4.nsx.lab.net/dev/disk.0.0 /tmp/tmpuVZnvv
INFO testec2docker4.nsx.lab.net.fs#1 btrfs filesystem label /tmp/tmpuVZnvv testec2docker4.nsx.lab.net.fs.1
INFO testec2docker4.nsx.lab.net.fs#1 btrfs subvol create /tmp/tmpuVZnvv/data
INFO testec2docker4.nsx.lab.net.fs#1 output:
Create subvolume '/tmp/tmpuVZnvv/data'
INFO testec2docker4.nsx.lab.net.fs#1 umount /tmp/tmpuVZnvv
INFO testec2docker4.nsx.lab.net.fs#1 provisioned
INFO testec2docker4.nsx.lab.net.fs#1 btrfs device scan
INFO testec2docker4.nsx.lab.net.fs#1 output:
Scanning for Btrfs filesystems
INFO testec2docker4.nsx.lab.net.fs#1 mount -t btrfs -o defaults,subvol=data LABEL=testec2docker4.nsx.lab.net.fs.1 /srv/testec2docker4.nsx.lab.net/data
INFO testec2docker4.nsx.lab.net.fs#0 mount -t btrfs -o subvolid=0 /var/lib/opensvc/testec2docker4.nsx.lab.net/dev/disk.0.0 /tmp/tmpNPV_d8
INFO testec2docker4.nsx.lab.net.fs#0 btrfs subvol create /tmp/tmpNPV_d8/docker
INFO testec2docker4.nsx.lab.net.fs#0 output:
Create subvolume '/tmp/tmpNPV_d8/docker'
INFO testec2docker4.nsx.lab.net.fs#0 umount /tmp/tmpNPV_d8
INFO testec2docker4.nsx.lab.net.fs#0 provisioned
INFO testec2docker4.nsx.lab.net.fs#0 btrfs device scan
INFO testec2docker4.nsx.lab.net.fs#0 output:
Scanning for Btrfs filesystems
INFO testec2docker4.nsx.lab.net.fs#0 mount -t btrfs -o defaults,subvol=docker LABEL=testec2docker4.nsx.lab.net.fs.1 /srv/testec2docker4.nsx.lab.net/docker
INFO testec2docker4.nsx.lab.net.container#0 starting docker daemon
INFO testec2docker4.nsx.lab.net.container#0 docker -H unix:///var/lib/opensvc/testec2docker4.nsx.lab.net/docker.sock -r=false -d -g /srv/testec2docker4.nsx.lab.net/docker -p /var/lib/opensvc/testec2docker4.nsx.lab.net/docker.pid --ip 10.0.0.221 --exec-opt native.cgroupdriver=cgroupfs
INFO testec2docker4.nsx.lab.net.container#0 docker -H unix:///var/lib/opensvc/testec2docker4.nsx.lab.net/docker.sock run -t -i -d --name=testec2docker4.nsx.lab.net.container.0 --net=bridge -p 80:80 -v /etc/localtime:/etc/localtime:ro --cgroup-parent /testec2docker4.nsx.lab.net/container.docker/container.0 ubuntu:14.10 /bin/bash
WARNING testec2docker4.nsx.lab.net.container#0 command succesful but stderr:
Unable to find image 'ubuntu:14.10' locally
14.10: Pulling from ubuntu
6d370c930acc: Pulling fs layer
9a63f1e91c4c: Pulling fs layer
74364530838d: Pulling fs layer
dce38fb57986: Pulling fs layer
dce38fb57986: Pulling fs layer
dce38fb57986: Layer already being pulled by another client. Waiting.
74364530838d: Verifying Checksum
74364530838d: Download complete
dce38fb57986: Verifying Checksum
dce38fb57986: Download complete
dce38fb57986: Download complete
9a63f1e91c4c: Verifying Checksum
9a63f1e91c4c: Download complete
6d370c930acc: Verifying Checksum
6d370c930acc: Download complete
6d370c930acc: Pull complete
9a63f1e91c4c: Pull complete
74364530838d: Pull complete
dce38fb57986: Pull complete
dce38fb57986: Already exists
ubuntu:14.10: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:6341c688b4b0b82ec735389b3c97df8cf2831b8cb8bd1856779130a86574ac5c
Status: Downloaded newer image for ubuntu:14.10
INFO testec2docker4.nsx.lab.net.container#0 output:
bed299be99fabccf55087d0af1e9ebcf886158c5f83c32efd0819c457c579d03
INFO testec2docker4.nsx.lab.net.container#0 wait for container up status
INFO testec2docker4.nsx.lab.net.container#0 wait for container operational
INFO testec2docker4.nsx.lab.net.container#1 docker -H unix:///var/lib/opensvc/testec2docker4.nsx.lab.net/docker.sock run -t -i -d --name=testec2docker4.nsx.lab.net.container.1 -v /etc/localtime:/etc/localtime:ro --net=container:testec2docker4.nsx.lab.net.container.0 --cgroup-parent /testec2docker4.nsx.lab.net/container.docker/container.1 nginx:latest
WARNING testec2docker4.nsx.lab.net.container#1 command succesful but stderr:
Unable to find image 'nginx:latest' locally
latest: Pulling from nginx
843e2bded498: Pulling fs layer
8c00acfb0175: Pulling fs layer
426ac73b867e: Pulling fs layer
d6c6bbd63f57: Pulling fs layer
4ac684e3f295: Pulling fs layer
91391bd3c4d3: Pulling fs layer
b4587525ed53: Pulling fs layer
0240288f5187: Pulling fs layer
28c109ec1572: Pulling fs layer
063d51552dac: Pulling fs layer
d8a70839d961: Pulling fs layer
ceab60537ad2: Pulling fs layer
063d51552dac: Verifying Checksum
063d51552dac: Download complete
ceab60537ad2: Verifying Checksum
ceab60537ad2: Download complete
28c109ec1572: Verifying Checksum
28c109ec1572: Download complete
91391bd3c4d3: Verifying Checksum
91391bd3c4d3: Download complete
426ac73b867e: Verifying Checksum
426ac73b867e: Download complete
4ac684e3f295: Verifying Checksum
4ac684e3f295: Download complete
0240288f5187: Verifying Checksum
0240288f5187: Download complete
d6c6bbd63f57: Verifying Checksum
d6c6bbd63f57: Download complete
8c00acfb0175: Verifying Checksum
8c00acfb0175: Download complete
d8a70839d961: Verifying Checksum
d8a70839d961: Download complete
b4587525ed53: Verifying Checksum
b4587525ed53: Download complete
843e2bded498: Verifying Checksum
843e2bded498: Download complete
843e2bded498: Pull complete
8c00acfb0175: Pull complete
426ac73b867e: Pull complete
d6c6bbd63f57: Pull complete
4ac684e3f295: Pull complete
91391bd3c4d3: Pull complete
b4587525ed53: Pull complete
0240288f5187: Pull complete
28c109ec1572: Pull complete
063d51552dac: Pull complete
d8a70839d961: Pull complete
ceab60537ad2: Pull complete
Digest: sha256:9d0768452fe8f43c23292d24ec0fbd0ce06c98f776a084623d62ee12c4b7d58c
Status: Downloaded newer image for nginx:latest
INFO testec2docker4.nsx.lab.net.container#1 output:
3512b1265a540d74d4deb1598434e9be7ddc14252a85b94b372d81cb3a5a8b34
INFO testec2docker4.nsx.lab.net.container#1 wait for container up status
INFO testec2docker4.nsx.lab.net.container#1 wait for container operational
send /etc/opensvc/testec2docker4.nsx.lab.net.conf to collector ... OK
update /var/lib/opensvc/testec2docker4.nsx.lab.net.push timestamp ... OK
Docker Service on Amazon, Btrfs on Lvm¶
Template:
[DEFAULT]
env = TST
docker_data_dir = /srv/{svcname}/docker
docker_daemon_args = --storage-driver=btrfs
app = NSX
[disk#0]
type = amazon
volumes = <size=15>
[disk#1]
type = lvm
name = {svcname}
pvs = /var/lib/opensvc/{svcname}/dev/disk.0.0
[fs#1]
type = btrfs
mnt = /srv/{svcname}
dev = /dev/{svcname}/root
mnt_opt = defaults,subvol=root
vg = {svcname}
size = 14G
[fs#2]
type = btrfs
mnt = /srv/{svcname}/data
dev = /dev/{svcname}/root
mnt_opt = defaults,subvol=data
[fs#3]
type = btrfs
mnt = /srv/{svcname}/docker
dev = /dev/{svcname}/root
mnt_opt = defaults,subvol=docker
[container#0]
type = docker
image = ghcr.io/opensvc/pause
rm = true
[container#1]
type = docker
image = ubuntu:latest
netns = container#0
run_args = -v /srv/{svcname}/data:/data:rw
command = /bin/bash
Docker Service on Amazon, Btrfs on Md Raid¶
Template:
[DEFAULT]
env = TST
docker_data_dir = /srv/{svcname}/docker
docker_daemon_args = --storage-driver=btrfs
app = NSX
[disk#0]
type = amazon
volumes = <size=5> <size-5> <size-5>
[disk#1]
type = md
uuid =
devs = /var/lib/opensvc/{svcname}/dev/disk.0.0 /var/lib/opensvc/{svcname}/dev/disk.0.1 /var/lib/opensvc/{svcname}/dev/disk.0.2
spares = 1
chunk = 1m
level = 1
[fs#1]
type = btrfs
mnt = /srv/{svcname}
dev = /dev/md/{svcname}.disk.1
mnt_opt = defaults,subvol=root
[fs#2]
type = btrfs
mnt = /srv/{svcname}/data
dev = /dev/md/{svcname}.disk.1
mnt_opt = defaults,subvol=data
[fs#3]
type = btrfs
mnt = /srv/{svcname}/docker
dev = /dev/md/{svcname}.disk.1
mnt_opt = defaults,subvol=docker
[container#0]
type = docker
image = ghcr.io/opensvc/pause
rm = true
Cluster-Ready HAProxy Service on Amazon¶
Single command provisioning:
om haproxy1.nsx.lab.net create --provision --leader --disable-rollback \
--resource '{"rtype": "DEFAULT", "nodes": "node12.nsx.lab.net", "docker_data_dir": "/srv/haproxy1.nsx.lab.net/docker", "env": "TST"}' \
--resource '{"rtype": "ip", "type": "amazon", "ipname": "<allocate>", "ipdev": "eth0", "docker_daemon_ip": true, "cascade_allocation": "ip#1.ipname"}' \
--resource '{"rtype": "ip", "ipdev": "eth0", "ipname": ""}' \
--resource '{"rtype": "disk", "type": "amazon", "volumes": "<size=5>"}' \
--resource '{"rtype": "fs", "type": "btrfs", "mnt_opt": "defaults,subvol=docker", "mnt": "/srv/haproxy1.nsx.lab.net/docker", "dev": "/var/lib/opensvc/haproxy1.nsx.lab.net/dev/disk.0.0"}' \
--resource '{"rtype": "fs", "type": "btrfs", "mnt_opt": "defaults,subvol=data", "mnt": "/srv/haproxy1.nsx.lab.net/data", "dev": "/var/lib/opensvc/haproxy1.nsx.lab.net/dev/disk.0.0"}' \
--resource '{"rtype": "container", "type": "docker", "image": "haproxy", "run_args": "-v /etc/localtime:/etc/localtime:ro -v /srv/haproxy1.nsx.lab.net/data:/data -p 80:80 -p 443:443 --net=bridge", "command": "haproxy -db -f /data/etc/haproxy.cfg"}'
Example haproxy.cfg file in /srv/haproxy1.nsx.lab.net/data/etc/haproxy.cfg
:
global
daemon
maxconn 256
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend http-in
bind *:80
default_backend servers
backend servers
server server1 10.0.0.60:8000 maxconn 32
server server1 10.0.0.61:8000 maxconn 32