Web server load balancing on (in the end) Raspberry Pi for less than 100 bucks


So…

The other day I got very interested in web server load balancing.

Again.

One of these things that most people would just never care about, but as always… I got obsessed with it.

Again.

Not so much because the technology is particularly complex (it is), or that you need to have so much knowledge of low level IP protocol to understand it well (you do). It was more that someone at work told me that I didn’t understand it. Such remarks alone have made smarter people than me insane; some of them brought elephants over the alps.

But, in an ancient workplace of mine, some 10-12 years ago, I ran into this topic for the first time. I was working at a company called Inserve, and a good old friend Rober Carlsson introduced me to the topic. Shortly after we started working together with a company called RadWare, selling their take on the issue. I can still remember how fascinated I was of their, at the time, so simple way of implementing so called “triangulation”, where a load balancer would get a TCP packet, change the MAC address in the headers and put it back on the network, so that a back end web-server would fetch it and send the reply directly over the router to the client.

I am getting way too detailed for this post already.

The other day, as I sad, I realized that I was a bit out of the loop. I had not researched the topic in quite a while. And all those company- and product- names I was using fluently a few years ago, was a bit faint. Extreme networks, F5, RadWare.

At work, we are using fairly old F5 boxes. They work well, and there are a couple of features I really enjoy. Setting up a load balanced web server environment is fairly straight forward.

  • Set up a virtual host
  • Set up a pool
  • Add back end members to the pool

Done.

And you can add rules to the pool (monitors), which are ran against all back end members. I if all monitors return OK, the targeted back end server deserves to stay in the configuration.

This is a very neat way of making sure that only correctly configured back end servers are serving web pages to the public.

A simple test can be to check if port 80 is responding. A more complex test would connect, request a web page, and parse the content for some keywords. This way, one can easily setup a web page that would represent an end to end view of the stack. I.e embedding a function in the web page, that connects to the back end database, checking for a certain value in the database. If all goes well, the web page would eventually show “Database connection OK”. A monitor could parse this output for this string, and voila, the test proves that both the web server and it’s database connectivity is ok.

I love such features.

But, and there is always a but. A major corporation can afford the big players. A clustered F5 system is expensive. The “brand new BMW” kind of expensive. They have really good products, don’t take me wrong, but it is not for the startup where a free lunch consists of half a sandwich.

So, I started looking around a bit and got fascinated by two products that I looked at a bit more:

  • http://Loadbalancer.org
  • http://zenloadbalancer.org

The first one is decently priced, but commercial. The second one, zenloadbalancer.org, is free as in beer.

I started looking into the zenloadbalancer.org product, and downloaded their ISO image (the product is based on Debian). There was a good video available on Youtube, so I got started in no time at all.

The magic in this sauce, is the web-frontend that the guys at ZenLoadbalancer delivers. The load balancing itself is based on the open source project Pen, which is fairly heavy to get into. But with the help from the web gui, I had a working solution up and running in an hour or so.

Basically, my configuration was:

  1. A forward of port 8080 on my firewall to port 80 on my internal IP 192.168.2.49, which would be the virtual IP on my load balancer
  2. the virtual IP on eth0:1 on my load balancer
  3. the two web servers, 192.168.2.21 and 192.168.2.22, which are the two apache web servers I had configured on the “inside”.
I configured a monitor to check my back end web servers (more about this in a separate post). It worked. Not that I had expected anything else.
But then I had a chat with another friend, Karl GilĂ©n, who I hooked up with a couple of Raspberry Pi:s. We started talking about a “Raspberry Pi only” clustered web farm. With a load balancer, a couple of web servers, perhaps also the mysql database on Raspberry Pi. Well, all is possible since it is all in the Debian distro. But now it had to be done.
I got started thinking of how I could reproduce the load balancing on the Raspberry Pi, and after just a few seconds it came to me. “Just rip it off off the zenloadbalancer box!”.
Connecting to the zenloadbalancer, i did the following:

[ccne lines=”-1″]
root@kmg-zenlb-0001:~# ps -ef | grep pen
root 1045 1 0 13:52 ? 00:00:00 /usr/local/zenloadbalancer/app/pen/bin/pen -S 10 -c 2049 -x 257 -F /usr/local/zenloadbalancer/config/Kalle_pen.cfg -C 127.0.0.1:10448 192.168.2.49:80
root 1055 1 0 13:52 ? 00:00:00 /usr/local/zenloadbalancer/app/pen/bin/pen -S 10 -c 2049 -x 257 -F /usr/local/zenloadbalancer/config/1200kcaltcp_pen.cfg -C 127.0.0.1:12162 192.168.2.48:80
root 27021 12111 0 17:30 pts/0 00:00:00 grep pen
root@kmg-zenlb-0001:~# cat /usr/local/zenloadbalancer/config/Kalle_pen.cfg
# Generated by pen 2012-09-21 13:43:20
# pen -S 10 -c 2049 -x 257 -F ‘/usr/local/zenloadbalancer/config/Kalle_pen.cfg’ -C 127.0.0.1:10448 192.168.2.49:80
no acl 0
no acl 1
no acl 2
no acl 3
no acl 4
no acl 5
no acl 6
no acl 7
no acl 8
no acl 9
acl 9 deny 0.0.0.0 0.0.0.0
no ascii
blacklist 30
no block
client_acl 0
control_acl 0
debug 0
no delayed_forward
no hash
no http
no log
no roundrobin
server 0 acl 0 address 192.168.2.21 port 80 max 0 hard 0
server 1 acl 0 address 192.168.2.22 port 80 max 0 hard 0
server 2 acl 0 address 0.0.0.0 port 0 max 0 hard 0
server 3 acl 0 address 0.0.0.0 port 0 max 0 hard 0
server 4 acl 0 address 0.0.0.0 port 0 max 0 hard 0
server 5 acl 0 address 0.0.0.0 port 0 max 0 hard 0
server 6 acl 0 address 0.0.0.0 port 0 max 0 hard 0
server 7 acl 0 address 0.0.0.0 port 0 max 0 hard 0
server 8 acl 0 address 0.0.0.0 port 0 max 0 hard 0
server 9 acl 0 address 0.0.0.0 port 0 max 0 hard 0
no stubborn
timeout 5
tracking 0
no web_stats
no weight
no prio
root@kmg-zenlb-0001:~# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:0c:29:62:9c:e1
inet addr:192.168.2.47 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe62:9ce1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:50817 errors:0 dropped:0 overruns:0 frame:0
TX packets:44155 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:32617453 (31.1 MiB) TX bytes:3929003 (3.7 MiB)

eth0:1 Link encap:Ethernet HWaddr 00:0c:29:62:9c:e1
inet addr:192.168.2.49 Bcast:192.168.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
[/ccne]

This could not be hard to reproduce on the raspberry pi, no? I brought down the IP address 192.168.2.49 on the zenloadbalancer box and just did the following on my Raspberry Pi.

[ccne lines=”-1″]
pi@raspberrypi /tmp $ sudo apt-get install pen
pi@raspberrypi /tmp $ vi /tmp/kalle.cfg
pi@raspberrypi /tmp $ cat /tmp/kalle.cfg
# Generated by pen 2012-09-21 13:43:20
# pen -S 10 -c 2049 -x 257 -F ‘/usr/local/zenloadbalancer/config/Kalle_pen.cfg’ -C 127.0.0.1:10448 192.168.2.49:80
no acl 0
no acl 1
no acl 2
no acl 3
no acl 4
no acl 5
no acl 6
no acl 7
no acl 8
no acl 9
acl 9 deny 0.0.0.0 0.0.0.0
no ascii
blacklist 30
no block
client_acl 0
control_acl 0
debug 0
no delayed_forward
no hash
no http
no log
no roundrobin
server 0 acl 0 address 192.168.2.21 port 80 max 0 hard 0
server 1 acl 0 address 192.168.2.22 port 80 max 0 hard 0
server 2 acl 0 address 0.0.0.0 port 0 max 0 hard 0
server 3 acl 0 address 0.0.0.0 port 0 max 0 hard 0
server 4 acl 0 address 0.0.0.0 port 0 max 0 hard 0
server 5 acl 0 address 0.0.0.0 port 0 max 0 hard 0
server 6 acl 0 address 0.0.0.0 port 0 max 0 hard 0
server 7 acl 0 address 0.0.0.0 port 0 max 0 hard 0
server 8 acl 0 address 0.0.0.0 port 0 max 0 hard 0
server 9 acl 0 address 0.0.0.0 port 0 max 0 hard 0
no stubborn
timeout 5
tracking 0
no web_stats
no weight
no prio
pi@raspberrypi /tmp $ sudo ifconfig eth0:1 192.168.2.49 netmask 255.255.255.0
pi@raspberrypi /tmp $ sudo pen -S 10 -c 2049 -x 257 -F /tmp/kalle.cfg -C 127.0.0.1:10448 192.168.2.49:80
[/ccne]

And… It works! I now run my site 1200kcal.com on port 8080 load balanced on a Raspberry Pi. If you are lucky, it is still running (I don’t aim to run this config for very long).

Ask around for how much your fellow web admins paid for their web load balancing (a lot), and compare it to my 35 chf Raspberry Pi, with a 10 chf power supply. My load balancer maxes out quite easily, though. =)