Services Lxc clusterisés sur OVH

Introduction

OVH propose des serveurs puissants et raccordés à internet pour une somme modique, et un mécanisme de bascule des IPv4. C'est une combinaison très intéressante pour les services clusterisés par OpenSVC. Ce guide explique les étapes nécessaires à l'intégration d'un tel cluster, avec des services de type LXC sur disques locaux. Ce type de services permet une bonne isolation entre services sans compromis sur les performance et la consommation de mémoire.

Préparation du noeud

Avant de passer à l'étape suivante, vous devez avoir un couple de serveurs mis à disposition par OVH, configurés en Debian Squeeze, avec leur noyau et leur lanceurs supportant LXC. Vous devez également avoir alloué une 'IP failover'. Enfin, l'agent OpenSVC doit être installé sur les deux noeuds (doc)

Paquets additionels

Install:

apt-get install lxc bridge-utils python2.6 python2.5 debootstrap rsync lvm2 ntp python-soappy

And opensvc from https://repo.opensvc.com/deb/

Bridge ethernet

Create a backend bridge connected to a dummy interface. In /etc/network/interfaces add the following block and activate the bridge using ifup br0:

auto br0
iface br0 inet static
    bridge_ports dummy0
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 5
    address 192.168.0.1
    netmask 255.255.255.0
    pre-up /sbin/modprobe dummy

Paramètres noyau

In /etc/sysctl.conf set the following parameters and reload the configuration using sysctl -p:

# lxc routing
net.ipv4.ip_forward=1
net.ipv4.conf.br0.proxy_arp=1

Configuration de cgroup

In /etc/fstab add the following line:

none /cgroup cgroup defaults 0 0

Then:

mkdir /cgroup
mount /cgroup

Préparation du service

Configuration des disques

OVH servers come with a 4 GB root filesystem, a ~4 GB swap partition and the rest of the disk is allocated to /home. The /home filesystem can be replaced by a single physical volume. Create a volume group over this pv and one or a set of logical volumes for each container. Format the logical volumes using the filesystem that suits you. Mount the logical volume set of the first container to create:

umount /home
vi /etc/fstab   # remove the /home entry
pvcreate /dev/your_home_dev
vgcreate vg0 /dev/your_home_dev
lvcreate -n service_name -L 20G vg0
mkfs.ext4 /dev/vg0/opt/opensvc_name
mkdir /opt/opensvc_name
mount /dev/vg0/opt/opensvc_name /opt/opensvc_name

Création du container

Prepare the lxc container creation wrapper:

gzip -dc /usr/share/doc/lxc/examples/lxc-debian.gz >/tmp/lxc-debian

Create the container rootfs:

/tmp/lxc-debian -p /opt/opensvc_name

Configuration de base d'un container

  • réseau

  • locale
  • tz
  • hosts
  • rc.sysinit (retirer les actions sur le swap et udev)

Créer le container

create a lxc config file as /tmp/lxc.conf containing:

lxc.utsname = service_name
lxc.tty = 4
lxc.pts = 1024
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.rootfs = /opt/opensvc_name/rootfs
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 254:0 rwm

and create the container with:

lxc-create -f /tmp/lxc.conf -n service_name

Start the container:

lxc-start -n service_name

Création du service OpenSVC

Trust the node root account to ssh-login into the container:

mkdir /opt/opensvc_name/rootfs/root/.ssh
cat /root/.ssh/id_dsa.pub >>/opt/opensvc_name/rootfs/root/.ssh/authorized_keys

Create the service configuration file:

[default]
app = MYAPP
vm_name = service_name
mode = lxc
service_type = PRD
nodes = node1.mydomain node2.mydomain
autostart_node = node1.mydomain
drpnode =

[fs#1]
dev = /dev/mapper/vg0-service_name
mnt = /opt/opensvc_name
mnt_opt = defaults
type = ext4
always_on = nodes

[ip#1]
ipdev = br0
ipname = service_name
post_start = /etc/opensvc/opensvc_name.d/ovh_routes start service_name 1.2.3.4
pre_stop = /etc/opensvc/opensvc_name.d/ovh_routes stop service_name 1.2.3.4

[sync#0]
src = /opt/opensvc_name/
dst = /opt/opensvc_name
dstfs = /opt/opensvc_name
target = nodes
snap = true

Routage OVH et bascule d'ip

create the trigger scripts store, which is synchronized across nodes:

mkdir -p /etc/opensvc/opensvc_name.dir
cd /etc/opensvc/
ln -s opensvc_name.dir opensvc_name.d

create and adapt the trigger scripts as /etc/opensvc/opensvc_name.dir/ovh_routes:

#!/bin/bash

svc=$2
vip=$3

route="$vip dev br0"

function has_route {
        ip route ls | grep "$route" >/dev/null >&1
}

case $1 in
start)
        has_route || ip route add $route
        /etc/opensvc/etc/$svc.d/ipfailover
        # make sure proxy_arp and ip_forwarding settings are set
        sysctl -p >/dev/null 2>&1
        # containers are not able to load kernel modules.
        # trigger loading of common ones from here
        iptables -L -n >/dev/null 2>&1
        ;;
stop)
        has_route && ip route del $route
        ;;
esac

and /etc/opensvc/opensvc_name.dir/ipfailover:

#!/usr/bin/python2.5

vip = '1.2.3.4'

nodes_ip = {
 'n2': dict(
    otheracc='ksXXXXX.kimsufi.com',
    thisip='a.b.c.d'),
 'n1': dict(
    otheracc='ksYYYYY.kimsufi.com',
    thisip='d.c.b.a'),
}

# login information
nic = 'xxxx-ovh'
password = 'xxxx'

#
# don't change below
#
from SOAPpy import WSDL
import sys

soap = WSDL.Proxy('https://www.ovh.com/soapi/ovh.wsdl')

try:
    session = soap.login( nic, password )
except:
    print >>sys.stderr, "Error login"

from os import uname
x, nodename, x, x, x = uname()

# dedicatedFailoverUpdate
try:
    result = soap.dedicatedFailoverUpdate(session,
                                       nodes_ip[nodename]['otheracc'],
                                       vip,
                                       nodes_ip[nodename]['thisip']);
    print "dedicated Failover Update successfull";
except:
    print >>sys.stderr, "Error dedicated Failover Update"

# logout
try:
    result = soap.logout( session )
except:
    print >>sys.stderr, "Error logout"

S'assurer que ce dernier script est propriété de root et qu'il a les permissions 700, puisqu'il contient des informations sensibles du point de vue de la sécurité.