Lxc clustered services on OVH

Introduction

OVH provides powerful internet-connected servers at an affordable price and a scriptable IPv4 takeover. This is a great combination for clustered services driven by opensvc. This cookbook explains the steps involved in integrating such a cluster with LXC services on a local disk to gain a decent partitioning between services without compromise on performance and memory usage.

Preparing a node

Before moving on to the next step, you should have a couple of servers delivered by OVH, setup with Debian Squeeze, which has initscripts and kernel adapted for LXC. You should also have an 'IP failover' available. Finally, the OpenSVC agent should be installed on both nodes (doc)

Additional packages

Install:

apt-get install lxc bridge-utils python2.6 python2.5 debootstrap rsync lvm2 ntp python-soappy

And opensvc from https://repo.opensvc.com/deb/

Ethernet bridge

Create a backend bridge connected to a dummy interface. In /etc/network/interfaces add the following block and activate the bridge using ifup br0:

auto br0
iface br0 inet static
    bridge_ports dummy0
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 5
    address 192.168.0.1
    netmask 255.255.255.0
    pre-up /sbin/modprobe dummy

Kernel parameters

In /etc/sysctl.conf set the following parameters and reload the configuration using sysctl -p:

# lxc routing
net.ipv4.ip_forward=1
net.ipv4.conf.br0.proxy_arp=1

Cgroup setup

In /etc/fstab add the following line:

none /cgroup cgroup defaults 0 0

Then:

mkdir /cgroup
mount /cgroup

Preparing the service

Disk setup

OVH servers come with a 4 GB root filesystem, a ~4 GB swap partition and the rest of the disk is allocated to /home. The /home filesystem can be replaced by a single physical volume. Create a volume group over this pv and one or a set of logical volumes for each container. Format the logical volumes using the filesystem that suits you. Mount the logical volume set of the first container to create:

umount /home
vi /etc/fstab   # remove the /home entry
pvcreate /dev/your_home_dev
vgcreate vg0 /dev/your_home_dev
lvcreate -n service_name -L 20G vg0
mkfs.ext4 /dev/vg0/opt/opensvc_name
mkdir /opt/opensvc_name
mount /dev/vg0/opt/opensvc_name /opt/opensvc_name

Container creation

Prepare the lxc container creation wrapper:

gzip -dc /usr/share/doc/lxc/examples/lxc-debian.gz >/tmp/lxc-debian

Create the container rootfs:

/tmp/lxc-debian -p /opt/opensvc_name

Basic container setup

• network
• locale
• tz
• hosts
• rc.sysinit (remove swaps and udev actions)

Create the container

create a lxc config file as /tmp/lxc.conf containing:

lxc.utsname = service_name
lxc.tty = 4
lxc.pts = 1024
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.rootfs = /opt/opensvc_name/rootfs
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 254:0 rwm

and create the container with:

lxc-create -f /tmp/lxc.conf -n service_name

Start the container:

lxc-start -n service_name

Opensvc service creation

Trust the node root account to ssh-login into the container:

mkdir /opt/opensvc_name/rootfs/root/.ssh
cat /root/.ssh/id_dsa.pub >>/opt/opensvc_name/rootfs/root/.ssh/authorized_keys

Create the service configuration file:

[default]
app = MYAPP
vm_name = service_name
mode = lxc
env = PRD
nodes = node1.mydomain node2.mydomain
orchestrate = start

[fs#1]
dev = /dev/mapper/vg0-service_name
mnt = /opt/opensvc_name
mnt_opt = defaults
type = ext4
standby = true

[ip#1]
ipdev = br0
ipname = service_name
post_start = /etc/opensvc/opensvc_name.d/ovh_routes start service_name 1.2.3.4
pre_stop = /etc/opensvc/opensvc_name.d/ovh_routes stop service_name 1.2.3.4

[sync#0]
src = /opt/opensvc_name/
dst = /opt/opensvc_name
dstfs = /opt/opensvc_name
target = nodes
snap = true

OVH routing and ipfailover

create the trigger scripts store, which is synchronized across nodes:

mkdir -p /etc/opensvc/opensvc_name.dir
cd /etc/opensvc/
ln -s opensvc_name.dir opensvc_name.d

create and adapt the trigger scripts as /etc/opensvc/opensvc_name.dir/ovh_routes:

#!/bin/bash

svc=$2
vip=$3

route="$vip dev br0"

function has_route {
        ip route ls | grep "$route" >/dev/null >&1
}

case $1 in
start)
        has_route || ip route add $route
        /etc/opensvc/etc/$svc.d/ipfailover
        # make sure proxy_arp and ip_forwarding settings are set
        sysctl -p >/dev/null 2>&1
        # containers are not able to load kernel modules.
        # trigger loading of common ones from here
        iptables -L -n >/dev/null 2>&1
        ;;
stop)
        has_route && ip route del $route
        ;;
esac

and /etc/opensvc/opensvc_name.dir/ipfailover:

#!/usr/bin/python2.5

vip = '1.2.3.4'

nodes_ip = {
 'n2': dict(
    otheracc='ksXXXXX.kimsufi.com',
    thisip='a.b.c.d'),
 'n1': dict(
    otheracc='ksYYYYY.kimsufi.com',
    thisip='d.c.b.a'),
}

# login information
nic = 'xxxx-ovh'
password = 'xxxx'

#
# don't change below
#
from SOAPpy import WSDL
import sys

soap = WSDL.Proxy('https://www.ovh.com/soapi/ovh.wsdl')

try:
    session = soap.login( nic, password )
except:
    print >>sys.stderr, "Error login"

from os import uname
x, nodename, x, x, x = uname()

# dedicatedFailoverUpdate
try:
    result = soap.dedicatedFailoverUpdate(session,
                                       nodes_ip[nodename]['otheracc'],
                                       vip,
                                       nodes_ip[nodename]['thisip']);
    print "dedicated Failover Update successfull";
except:
    print >>sys.stderr, "Error dedicated Failover Update"

# logout
try:
    result = soap.logout( session )
except:
    print >>sys.stderr, "Error logout"

Make sure this last script is owned by root and has 700 permissions, as it contains important credentials.