Create a Software Load Balancer w/ Content Switching and SSL

I was posed with a problem at work. We have a hardware load-balancer in our production environment (Cisco ACE). We were required to build out a disaster recovery site without the funds to purchase a hardware load-balancer. I then began my research to make it happen. I had a really hard time finding a document for this that was end-to-end, so I decided to write this.

This was originally created back in 2011 so some of the software versions may be a little behind. You can try newer versions if you like, the process should be fairly similar.

The load-balancer software is called HAproxy and it runs on Linux systems. You can install it on any distribution that you like, but for this guide I used Ubuntu Server 9.10. I cant guarantee the same results from a different distro.

Install the OS

  1. Download Ubuntu server 9.10 and install it as a VM or a physical server.
  2. Make sure you have correct IP settings so the server can access the internet, or else apt will not be able to install properly.
  3. When the software package installation screen comes up, only install OpenSSH server.
  4. Log in as the user you specified at installation.
  5. Enter the su -i command to gain privileges to reset the root password
  6. Enter passwd root to change the password to something specific.

Install HAProxy

  1. Once you get your OS up and running you need to install HAproxy.
  2. Install HAproxy with apt. Run the following command (you have to be logged in as root or use su -i)
  • apt-get install haproxy
  1. Back up the old config file
    • cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxyORIGINAL.cfg
  2. Paste the following into your config file and build off of it (this contains content switching)
global
   log 127.0.0.1 local0
   maxconn 4096
   uid 99
   gid 99
   daemon

defaults
   mode http
   log global
   option tcplog
   option httpclose
   retries 3
   maxconn 2000
   contimeout      5000
   clitimeout      50000
   srvtimeout      50000

frontend LB1 *:80
   option forwardfor
   reqadd          X-Forwarded-Proto:\ https
   reqadd          FRONT_END_HTTPS:\ on
   acl FARM1-acl url_sub -i Hello
   acl FARM2-acl url_sub -i Goodbye
   use_backend Hello if FARM1-acl
   use_backend Goodbye if FARM2-acl
   default_backend DEFAULT

backend FARM1
   stats enable
   stats auth admin:password10
   balance roundrobin
   #option ssl-hello-chk
   server SERVER1 10.0.5.1:80 check
   server SERVER2 10.0.5.2:80 check

backend FARM2
   stats enable
   stats auth admin:password10
   balance roundrobin
   #option ssl-hello-chk
   server SERVER3 10.0.5.3:80 check
   server SERVER4 10.0.5.4:80 check

backend DEFAULT
   stats enable
   stats auth admin:password10
   balance roundrobin
   #option ssl-hello-chk
   server SERVER5 10.0.5.5:80 check
   server SERVER6 10.0.5.6:80 check
  1. Now set HAProxy to start on reboot.
    • vim /etc/default/haproxy
    • Change the ENABLED = 0 to 1
  2. Restart the service and make sure you can hit your frontend IP on port 80.
    • service haproxy restart
  3. When you make changes to the /etc/haproxy.cfg file make sure to run service haproxy restart


Install Stunnel

Stunnel handles the SSL backend connections. Without Stunnel, content switching would not be possible as the traffic would be encrypted.
  1. Create a directory to download stunnel and it’s patch to
    • mkdir /root/stunnel
    • cd /root/stunnel
  2. Download Stunnel and it’s patch (in this version i used version 4.32)
  3. Patch Stunnel
    • cd stunnel-4.32
    • patch -p1 < ../stunnel-4.32-xforwarded-for.diff
  4. Install prerequisites
    • apt-get install libcurl3-openssl-dev
  5. Install stunnel
    • ./configure
    • make && make install

Generate Self-signed Certs

This step is for TESTING ONLY. Do this if if you dont have your real certs yet. To verify everything you’ll need a certificate and private key.
  1. Install OpenSSL:
    • apt-get install openssl
  2. Make a certificate directory:
    • mkdir /etc/certs
    • cd /etc/certs
  3. Generate Private key:
    • openssl genrsa -out server.key 1024
  4. Generate CSR:
    • openssl req -new -key server.key -out server.csr
  5. Generate Self-signed cert
    • openssl x509 -req -days 1200 -in server.csr -signkey server.key -out server.crt
  6. Make the stunnel config file
    • mkdir /etc/stunnel
    • vim /etc/stunnel/stunnel.conf
  7. Add the following to the basic config for stunnel:
cert=/etc/certs/server.crt
key = /etc/certs/server.key
;setuid = nobody
;setgid = nogroup

pid = /etc/stunnel/stunnel.pid
debug = 3
output = /etc/stunnel/stunnel.log

socket=l:TCP_NODELAY=1
socket=r:TCP_NODELAY=1

[https]
accept=192.168.1.1:443
connect=192.168.1.1:80
TIMEOUTclose=0
xforwardedfor=yes
  1. Set stunnel to start on reboot
    • vim /etc/init.d/stunnel
    • Paste the following into that file:
#!/bin/bash
#
# stunnel      This shell script takes care of starting and stopping
#              stunnel
#
# chkconfig: 345 80 30
# description:  Secure tunnel

# processname: stunnel
# config: /etc/stunnel/stunnel.conf
# pidfile: /var/run/stunnel/stunnel.pid

# Source function library.
. /lib/lsb/init-functions

# Source stunnel configureation.
if [ -f /etc/sysconfig/stunnel ] ; then
 . /etc/sysconfig/stunnel
fi

RETVAL=0
prog="stunnel"

start() {
 # Start daemons.

 echo -n $"Starting $prog: "
 if test -x /usr/local/bin/stunnel ; then
   /usr/local/bin/stunnel /etc/stunnel/stunnel.conf
 fi
 RETVAL=$?
 echo
 [ $RETVAL -eq 0 ] && touch /var/lock/stunnel
 return $RETVAL
}

stop() {
 # Stop daemons.
 echo -n $"Shutting down $prog: "
 killproc stunnel
 RETVAL=$?
 echo
 [ $RETVAL -eq 0 ] && rm -f /var/lock/stunnel
 return $RETVAL
}

# See how we were called.
case "$1" in
  start)
 start
 ;;
  stop)
 stop
 ;;
  restart)
 stop
 start
 RETVAL=$?
 ;;
  condrestart)
 if [ -f /var/lock/stunnel ]; then
     stop
     start
     RETVAL=$?
 fi
 ;;
  status)
 status stunnel
 RETVAL=$?
 ;;
  *)
 echo $"Usage: $0 {start|stop|restart|condrestart|status}"
 exit 1
esac

exit $RETVAL
  1. Save it and set it so it can be executed
    • chmod 755 /etc/init.d/stunnel
  2. Test the service
    • Service stunnel stop
    • Service stunnel start
    • (you might need to run the start command a couple times before it works)
  3. Install chkconfig so the service can start on it’s own
    • apt-get install chkconfig
    • chkconfig stunnel on
  4. Restart the server for good measure and make sure your services start.
  5. Finally test the function of HAproxy by going to http://host and https://host with the respective URL paths to test content switching.

Export Real Certificate

When you are done testing you will want to use the actual certificate for the servers that you are load-balancing for. In my case the real certificate was on a windows box. I had to export the certificate and private key from IIS and import it into something stunnel could understand.
  1. Go to the IIS server that has the certificate.
  2. Browse to the site that has the cert and right click it then go to properties.
  3. Click the Directory Security tab and click View Certificate…
  4. Click Details → Copy to File…
  5. Click Yes, export the private key
  6. Uncheck enable strong protection and click next. Make sure all sub boxes are not checked.
  7. Leave the password fields blank
  8. Pick the path where you want to save it.
  9. Click Next → Finish
  10. Take that file and copy it to your load balancer
  11. Once it is copied over cd to the directory that you moved it to and run the following command:
    • openssl pkcs12 -in mycertificate.pfx -nodes -out CERT.pem
  12. Generate your private key associated with the cert.
    • openssl rsa CERT.pem -out mycertificate.key
  13. Edit your /etc/stunnel.conf file to reflect the changes. E.g.
    • cert=/etc/certs/CERT.pem
  14. Restart stunnel

Enable Logging

  1. Install syslog if it is not already installed
    • apt-get install syslogd
  2. Change the config of syslog to allow socket connections
    • vim /etc/default/syslogd
    • SYSLOGD=”-r”
  3. Set the path to log to
    • vim /etc/syslog.conf
    • local0.* /var/log/messages
  4. Restart syslog
    • /etc/init.d/sysklogd restart
  5. Make sure haproxy is set up to log
global
  log 127.0.0.1 local0

defaults
  log global
  option tcplog

 For Servers that have Backend SSL’s

Use nginx. The load-balancing algorithm isnt as effictive as HAProxy, but HAProxy does not support servers on the backend with SSL certificates. This example uses the same base version of ubuntu server with SSH server installed only.
  1. Install nginx
    • apt-get install nginx
      OR
    • Install from source:
./configure  --conf-path=/etc/nginx/nginx.conf --with-http_dav_module  --with-http_ssl_module  --with-http_stub_status_module  --add-module=/root/http-healthcheck  --add-module=/root/upstream

aptitude show nginx
apt-get install libc6
apt-get install libgd2-noxpm
apt-get install libgd2-xpm libgeoip1 libpcre3 libssl0.9.8 libxml2 libxslt1.1 zlib1g lsb-base
  1. Install chkconfig
    • apt-get install chkconfig
  2. See if nginx starts with the default config
    • service nginx restart
  3. Make sure you have your SSL certificate and key ready. Without it the next steps arent going to work. Refer the above section on “Export Real SSL
  4. Modify the default nginx config. This example uses content filtering with front-end SSL proxying.
user www-data;
worker_processes  2; #this server has two processors
worker_cpu_affinity 01 10; #assigning each worker to a processor

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;

    access_log  /var/log/nginx/access.log;

    sendfile        on;

    keepalive_timeout  65;
    tcp_nodelay        on;

    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;

#defining the farms and backend ports
upstream DEFAULT {
  server 10.0.5.5:443;
  server 10.0.5.6:443;
 }

upstream FARM1 {
  server 10.0.5.1:443;
  server 10.0.5.2:443;
}

upstream FARM2 {
  server 10.0.5.3:443;
  server 10.0.5.4:443;
}

#define the server
server {

  listen 10.0.5.50:443;

  ssl on;
  ssl_certificate /etc/certs/CERT.pem; #location of cert
  ssl_certificate_key /etc/certs/KEY.key; #location of key

  server_name host.header.com;

#location is where content filtering happens. Each path after the / will get directed to the 
#farm defined below w/ the proxy_pass command
  location ~* ^/virtual1|virtual2|virtual3 {

   proxy_set_header  X-Real-IP  $remote_addr;

   proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header Host $http_host;
   proxy_redirect off;
   proxy_max_temp_file_size 0;

   proxy_pass https://VIRTUAL1;

  }

  location ~* ^/virtual4|virtual5 {

   proxy_set_header  X-Real-IP  $remote_addr;

   proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header Host $http_host;
   proxy_redirect off;
   proxy_max_temp_file_size 0;

   proxy_pass https://VIRTUAL5;

  }

  location ~* ^/virtual6|virtual7 {

   proxy_set_header  X-Real-IP  $remote_addr;

   proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header Host $http_host;
   proxy_redirect off;
   proxy_max_temp_file_size 0;

   proxy_pass https://VIRTUAL6;

  }

 }

}
  1. Restart nginx
    • service nginx restart
  2. Set nginx to start on reboot
    • chkconfig nginx on
  3. Make sure nginx is running properly with the netstat -lp command
 
root@LBHOST:~# netstat -lp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 LBHOST.local:https *:*                     LISTEN      1322/nginx
tcp        0      0 *:www                   *:*                     LISTEN      1322/nginx
tcp        0      0 *:ssh                   *:*                     LISTEN      882/sshd
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN      882/sshd
Advertisements
Create a Software Load Balancer w/ Content Switching and SSL

2 thoughts on “Create a Software Load Balancer w/ Content Switching and SSL

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s