Cluster Backend Networks¶
These networks are only required for services private ip auto-allocation. If configured, the cluster DNS exposes the allocated ip addresses as predictible names, and the cluster Ingress Gateways or portmapping can expose the services to clients outside the cluster.
OpenSVC relies on CNI for this subsystem. Any CNI plugin can be used but some plugins can have dependencies like etcd or consul, which OpenSVC does not require for himself. The bridge and weave plugin, having no such dependencies, are simpler to setup.
Install CNI¶
From package¶
Some distributions ship CNI packages.
On Red Hat or CentOS 7, for example, CNI is served by the EPEL repositories
# to activate epel repositories:
# yum install -y epel-release
yum install -y containernetworking-cni
Then tell OpenSVC where to find the CNI plugins and network configurations
om cluster set --kw cni.plugins=/usr/libexec/cni --kw cni.config=/var/lib/opensvc/cni/net.d
From upstream¶
cd /tmp
wget https://github.com/containernetworking/cni/releases/download/v0.6.0/cni-amd64-v0.6.0.tgz
wget https://github.com/containernetworking/plugins/releases/download/v0.6.0/cni-plugins-amd64-v0.6.0.tgz
sudo mkdir -p /opt/cni/bin
cd /opt/cni/bin
sudo tar xvf /tmp/cni-amd64-v0.6.0.tgz
sudo tar xvf /tmp/cni-plugins-amd64-v0.6.0.tgz
sudo mkdir -p /opt/cni/net.d
Here the plugins and network configurations directories are aligned with the OpenSVC defaults.
Configure networks¶
Networks are declared in the OpenSVC node or cluster configuration.
The agent create the CNI configuration files as needed.
Local Bridge¶
A local bridge network is always present and named default.
To create another network of this type, named local1, available on every cluster node
$ om cluster set --kw network#local1.type=bridge --kw network#local1.network=10.10.10.0/24
To create another network of this type, named local1, available on the current cluster node only:
$ om node set --kw network#local1.type=bridge --kw network#local1.network=10.10.10.0/24
Routed Bridge¶
This network type split the subnet into per-node segments. Trafic is routed from node-to-node via static routes to each segment, and ipip tunnels are created if necessary.
The simple bridge CNI plugin is used for IPAM and plumbing in network namespaces, and OpenSVC is responsible for node-to-node routing and tunneling.
To create a network of this type, named backend1, spanned on every cluster node
$ om cluster set --kw network#backend1.type=routed_bridge \
--kw network#backend1.network=10.11.0.0/16 \
--kw network#backend1.ips_per_node=1024
In this example, the network is split like:
- node 1 : 10.11.0.0/22
- node 2 : 10.11.4.0/22
- node 3 : 10.11.8.0/22
- ...
Tunnel endpoints addresses are guessed using a lookup of the nodenames. Different addresses can be setup if necessary, using:
$ om cluster set --kw network#backend1.addr@node1=1.2.3.4 \
--kw network#backend1.addr@node2=1.2.3.5 \
--kw network#backend1.addr@node3=1.2.4.4
Some hosting providers, like OVH, don't support static network routes from node to node, even if they have an ip address in a common subnet. For this situation, you can force OpenSVC to always use tunnels for this backend network:
$ om cluster set --kw network#backend1.tunnel=always
Weave¶
On each node:
sudo curl -L git.io/weave -o /usr/local/bin/weave
sudo chmod a+x /usr/local/bin/weave
The weave plugins runs daemons packaged as docker images. Before proceeding to the next weave installation step, make sure the docker daemon is started at boot and disable MountFlags.
Also make sure the OpenSVC Cluster is configured and joined before the next step, for the cluster.nodes reference to be resolved.
In this example, the package install plugins and config directories are used. Please adapt those paths as required.
On each node:
sudo sed -i s/^MountFlags=slave/#MountFlags=slave/ /lib/systemd/system/docker.service
sudo systemctl enable docker
sudo systemctl start docker
sudo weave setup
sudo weave launch $(om node get --kw cluster.nodes)
If CNI was installed from package, the weave plugin needs to be referenced in the package plugin dir:
cd /usr/libexec/cni/
sudo ln -s /opt/cni/bin/weave-ipam weave-ipam
sudo ln -s /opt/cni/bin/weave-net weave-net
Finally declare the network:
$ om cluster set --kw network#weave1.type=weave --kw network#weave1.network=10.32.0.0/12
Use in service configurations¶
Here is a typical ip resource configuration, using the "weave" CNI network configured above.
[ip#0]
type = cni
network = weave1
netns = container#0
expose = 80/tcp
The container pointed by netns can be a docker or lxc container. netns can also be left empty, causing the weave ip address to be assigned to the service cgroup.
The expose keyword is optional. If set, a SRV record is served by the cluster DNS (in this example _http._tcp.<svcname>.<namespace>.svc.<clustername>). If expose is set to portmapping expression, for example 80:8001/tcp, the portmap CNI plugin is will configure the portmapping and expose the 80/tcp backend server on the 8001 port of the node public ip addresses.
Useful commands¶
List networks:
# om net ls
default
lo
backend2
backend
Show networks status:
# om net status
name type network size used free pct
|- backend routed_bridge 10.40.0.0/16 65536 4 65532 0.01%
|- backend2 routed_bridge 10.41.0.0/16 65536 1 65535 0.00%
|- default bridge 10.22.0.0/16 65536 0 65536 0.00%
`- lo loopback 127.0.0.1/32 1 0 1 0.00%
Show a network detailled status:
# om net status --verbose --name backend
name type network size used free pct
`- backend routed_bridge 10.40.0.0/16 65536 4 65532 0.01%
`- ip node service resource
|- 10.40.0.10 c1n1 testbackend/svc/s1 ip#0
|- 10.40.12.7 c2n2 testbackend/svc/s4 ip#0
|- 10.40.4.9 c1n2 testbackend/svc/s2 ip#0
`- 10.40.8.7 c2n1 testbackend/svc/s3 ip#0