Sozu HA¶
Purpose¶
Sozu is a http reverse proxy from Clever Cloud, used by Cloud providers and PaaS provider to drive their moving Web infrastructure.
This survival guide describes how to deploy and manage a highly available Sozu setup, using OpenSVC cluster.
Prerequisites¶
Cluster¶
2 linux nodes Ubuntu 22.04 LTS
docker/podman
inotify-tools
Note
opensvc cluster installation is not covered in this howto
Installing¶
Deploy the sozu config map¶
A config map is used to store sozu related files like configuration, admin script.
om igw/cfg/sozu create
om igw/cfg/sozu add --key config.toml --from https://raw.githubusercontent.com/opensvc/opensvc_templates/main/sozu/config.toml
om igw/cfg/sozu add --key watch_directory.sh --from https://raw.githubusercontent.com/opensvc/opensvc_templates/main/sozu/watch_directory.sh
om igw/cfg/sozu add --key state.json
Deploy the sozu secret map¶
A secret is used to host the ssl certificates served by the sozu deployment.
om igw/sec/sozu create
om igw/sec/sozu add --key cert.pem --from /path/to/cert.pem
om igw/sec/sozu add --key key.pem --from /path/to/key.pem
om igw/sec/sozu add --key chain.pem --from /path/to/chain.pem
Note
Many certs can be added in this secret object. They will be available in the /certs folder inside the container
Deploy the sozu service¶
om igw/svc/sozu deploy --config https://raw.githubusercontent.com/opensvc/opensvc_templates/main/sozu/sozu.conf
How it works¶
The sozu HA deployment is an active/active opensvc service. One sozu instance per cluster node.
The initial configuration is loaded from config map key “config.toml”. The sozu configuration changes are stored in the state.json file on disk (automatic_state_save = true in config.toml file), which is copied back into the configmap object, and then replicated to other cluster nodes. Finally the remote sozu instances load the new state.json file to apply the changed configuration. This mecanism allow both instances to stay in sync.
Warning
this ha setup is working but has not been tested on large scale deployments
Checking Status¶
Cluster¶
Cluster status can be checked with command om mon
Threads demo1 demo2 daemon running | O collector running | O dns running hb#1.rx running [::]:10000 | / O hb#1.tx running | / O hb#2.rx running relay2.opensvc.com | / O hb#2.tx running | / O listener running :1214 monitor running scheduler running Nodes demo1 demo2 score | 70 70 load 15m | 0.0 0.0 mem | 9/98%:3.82g 9/98%:3.82g swap | - - state | */svc/* demo1 demo2 igw/svc/sozu up ha 2/2 | O^ O^ system/svc/vip up ha 1/1 | O^ X
Service¶
Service status can be checked with command om igw/svc/sozu print status
igw/svc/sozu up
`- instances
|- demo2 up idle, started
`- demo1 up idle, started
|- volume#cfg ........ up sozu-cfg
|- volume#scripts ........ up sozu-scripts
|- fs#flag ........ up fs.flag
|- container#0 ...../.. up docker google/pause
|- container#1 ...../.3 up docker clevercloud/sozu:d7b23c9fe877394cc3f2130d2fe5e76274dbf6c0
|- app#watch ...../.. up forking: watch_directory.sh
|- sync#i0 ..DO./.. n/a rsync svc config to nodes
`- task#stateload ...O./.. n/a docker clevercloud/sozu:d7b23c9fe877394cc3f2130d2fe5e76274dbf6c0
Note
add option -r to force immediate ressource status evaluation (om igw/svc/sozu print status -r)
Management commands¶
Starting service¶
om igw/svc/sozu start
Stopping service (all instances)¶
om igw/svc/sozu stop
Stopping service (1 instance)¶
om igw/svc/sozu stop --local
Fetching service config¶
om igw/svc/sozu print config
Editing service config¶
om igw/svc/sozu edit config
Listing config map keys¶
om igw/cfg/sozu keys
Editing config map key¶
om igw/cfg/sozu edit --key config.toml
Example¶
In the logs below, we can see that a config change on first node is replicated to the other sozu instance in a few seconds.
On node demo1
root@demo1:~# om igw/cfg/sozu decode --key state.json | wc -l
0
root@demo1:~# om igw/svc/sozu enter --rid container#1
OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown
/ # sozu -c /etc/sozu/config.toml cluster list
2024-07-01T14:07:24.203948Z 1719842844203948235 24 CTL INFO Ran the query successfully
Success: Ran the query successfully
┌────────────┬─────────────────────┬─────────────────────┬─────────────────────┬────────────────┐
│ cluster id │ worker 0 │ worker 1 │ worker main │ desynchronized │
├────────────┼─────────────────────┼─────────────────────┼─────────────────────┼────────────────┤
│ MyCluster │ 7980951202874738186 │ 7980951202874738186 │ 7980951202874738186 │ │
├────────────┼─────────────────────┼─────────────────────┼─────────────────────┼────────────────┤
│ TcpTest │ 4135859621253794451 │ 4135859621253794451 │ 4135859621253794451 │ │
└────────────┴─────────────────────┴─────────────────────┴─────────────────────┴────────────────┘
/ # sozu -c /etc/sozu/config.toml cluster add --id NEW_CLUSTER --load-balancing-policy round_robin
2024-07-01T14:07:55.387784Z 1719842875387784851 25 CTL INFO Successfully executed the request on all workers
Success: Successfully executed the request on all workers
No content
/ # sozu -c /etc/sozu/config.toml cluster list
2024-07-01T14:07:59.218130Z 1719842879218130199 26 CTL INFO Ran the query successfully
Success: Ran the query successfully
┌─────────────┬──────────────────────┬──────────────────────┬──────────────────────┬────────────────┐
│ cluster id │ worker 0 │ worker 1 │ worker main │ desynchronized │
├─────────────┼──────────────────────┼──────────────────────┼──────────────────────┼────────────────┤
│ NEW_CLUSTER │ 17225215009394938232 │ 17225215009394938232 │ 17225215009394938232 │ │
├─────────────┼──────────────────────┼──────────────────────┼──────────────────────┼────────────────┤
│ MyCluster │ 7980951202874738186 │ 7980951202874738186 │ 7980951202874738186 │ │
├─────────────┼──────────────────────┼──────────────────────┼──────────────────────┼────────────────┤
│ TcpTest │ 4135859621253794451 │ 4135859621253794451 │ 4135859621253794451 │ │
└─────────────┴──────────────────────┴──────────────────────┴──────────────────────┴────────────────┘
/ # exit
root@demo1:~# om igw/cfg/sozu decode --key state.json | wc -l
16
root@demo1:~#
On node demo2
root@demo2:~# om igw/svc/sozu enter --rid container#1
OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown
/ #
/ # sozu -c /etc/sozu/config.toml cluster list
2024-07-01T14:08:09.812256Z 1719842889812256391 24 CTL INFO Ran the query successfully
Success: Ran the query successfully
┌─────────────┬──────────────────────┬──────────────────────┬──────────────────────┬────────────────┐
│ cluster id │ worker 0 │ worker 1 │ worker main │ desynchronized │
├─────────────┼──────────────────────┼──────────────────────┼──────────────────────┼────────────────┤
│ NEW_CLUSTER │ 17225215009394938232 │ 17225215009394938232 │ 17225215009394938232 │ │
├─────────────┼──────────────────────┼──────────────────────┼──────────────────────┼────────────────┤
│ TcpTest │ 4135859621253794451 │ 4135859621253794451 │ 4135859621253794451 │ │
├─────────────┼──────────────────────┼──────────────────────┼──────────────────────┼────────────────┤
│ MyCluster │ 7980951202874738186 │ 7980951202874738186 │ 7980951202874738186 │ │
└─────────────┴──────────────────────┴──────────────────────┴──────────────────────┴────────────────┘
/ #