Cloud-Based Web Application Check-Script

production_inspection

Monitoring applications can be tricky. As a network engineer it’s important that the applications that I serve up are available and working properly. To network engineers it’s important that resources are up and running from the Internet. This script checks a variety of aspects pertaining to a public URL.

  • Availability (TCP ports)
  • Content matching (HTTP GET)
  • Certificate Expiration
  • Security (SSL disabled, proper chain, etc.)
  • DNS (cert name mismatch, DNS lookup)

The script runs from a Linux server I have in the cloud. Use any platform you choose. All that is required is Perl, curl, openssl and netcat.The MIME::Lite and DATE::Calc Perl modules are also required.

The script will run through a list of URL’s defined by you and check for content we’d expect when the application is up. We run through a lot of other tests mentioned above. Take a look at the script and see what it checks for.

The script will push output to an HTML file that allows for “user-friendly” appearance in email alerts and the HTML file can be uploaded to a webserver.

Configure the script to run periodically with a cron job.

This is a free, easily modified script that can help monitor your applications without much overhead. Use and modify to your specific needs.

#!/usr/bin/perl
use MIME::Lite;
use Date::Calc qw(
Date_to_Days);

#############################################################
# -----VERSION NOTES-----              
# 1.0: Base configuration
# 1.1: Added color/formatting changes to output
# 1.2: Added ncat port validation after failure
# 1.3: Added email functionality
# 1.4: Modified code to send HTML instead of text
# 1.5: Save HTML output to file and SFTP to dashboard server
# 1.6: Added additional logic for failure types, e.g. 404, etc.
# 2.0: Added Email logic for detecting failures
# 2.1: Added cert checking logic


#############################################################
# To test an application, add the full URL of the app you
# want to test to the apps variable. Secondly add a string of 
# content that matches when the application is up and running. 
# If the content doesn't match, the application is marked as 
# down. 
#############################################################

# Applications short or common name goes here e.g. 'Customer WebUI'
@shortname = (	#App1
				'Google',
				#App2
				'Fake Domain'
			);
			  
# FULL application URL goes here to test for availability and content
# !!!!!MAKE SURE YOU ADD THE PORT NUMBER AFTER THE HOSTNAME FOR THE TEST TO WORK!!!!!
@apps = (	'https://www.google.com:443/',
			'https://fakedomain.domains.net:443/'
		);

# Strings to match against when you know the application is up go here. 
@validator = (	'Google Search',
				'Bad'
			);
			  
# Count number of variables in array to loop can run properly
$numapps = (scalar @apps);

# Define variables

# Needed for our while statement
$a = 0;
# If one failure check passed, don't check for other failures. Reduces redundant information.
$b = 0;
# If c = 1, app is down. Used for e-mail notifications
$c = 0;

# Define HTML array to be sent via. SFTP
my @html = ();

# Set datetime
$time = `TZ=":US/Eastern" date`;

# Add environmental header in HTML file
push @html, '<head><title>App Status</title></head>';
push @html, '

<h1>Application Status</h1>


';
push @html, '

<h3>The applications listed here are checked once every 30 minutes</h3>


';
push @html, '

Last Checked: ' ;
push @html, $time;
push @html, '

';

# Tell the user that something is happening
print "The test is running, please wait... \n";

# Begin the checks
while($a < $numapps){

 # Grab URL components to allow for more segmented testing, use variables as needed
 my($protocol, $host, $port, $uri) = $apps[$a] =~ m|(.*)://([a-zA-Z0-9\-.]+):([0-9]+)?(.*)?|;

 # Lets run some quick security tests first
 $sslv2check = `timeout 3 openssl s_client -connect '$host':443 -ssl2 2>/dev/null`;
 $sslv3check = `timeout 3 openssl s_client -connect '$host':443 -ssl3 2>/dev/null`;

 #********************
 #* CERT DATE LOGIC *
 #********************
 
 # Get todays year, month and date
 $year = `TZ=":UTC" date +%Y`;
 $month = `TZ=":UTC" date +%m`;
 $date = `TZ=":UTC" date +%d`;
 
 #Convert todays date to a numeric date count
 $now = Date_to_Days($year,$month,$date);
 
 # Grab certificate date
 $cert = `timeout 3 openssl s_client -connect $host:443 -tls1 2>/dev/null| openssl x509 -noout -enddate 2>/dev/null| cut -f2 -d'=' | xargs -0 date +%F -d 2>/dev/null`;

 # Split the output into DATE MONTH DAY
 my @certdates = split('-', $cert); 


 #End certificate expiration logic
 
 if ($sslv3check =~ /Server public key is/) {
 $sslv3check = 1;
 print "We matched the SSLv3 security check\n";
 }
 if ($sslv2check =~ /Server public key is/) {
 $sslv2check = 1;
 print "We matched the SSLv2 security check\n";
 }

 # The standard check with a 60 second timeout. This is what grabs the HTTP information response from server
 $check = `curl '$apps[$a]' -m 60 2> stderr.txt`;
 
 # If the check failed, we want to preserve that information. Stderr is saved to a variable and the checks below are ran. 
 undef $stderr;
 my $stderr = `cat stderr.txt`;
 
 # Matched DNS unresolvable 
 if ($stderr =~ /curl: \(6{1}\)/) {
 
 # I set this here so we don't run other redundant checks.
 $b = 1;
 
 print "We matched the DNS stderr log\n";
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a>: <b>We couldn\'t resolve the host that you specified</b> using public DNS servers. Something is amiss...'; 
 }
 #Matched a TCP reset
 elsif ($stderr =~ /curl: \(56\)/) {
 
 $b = 1;
 
 print "We matched the TCP Reset rule\n";
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a>: <b>We were sent a reset!!</b> Since we did not see the construction page, there are a few reasons for this:<ul>';
 push @html, '<li>The server is not accepting requests at all. Check the server locally and make sure it does the same with localhost.</li>';
 push @html, '<li>The firewall is blocking us, DNS could be wrong, or the firewall isn\'t listening on this port</li></ul>'; 
 } 
 #Matched a cert name mismatch
 elsif ($stderr =~ /curl: \(51\)/) {
 
 undef $check;
 $check = `curl -k '$apps[$a]' -m 60 2> stderr.txt`;
 
 print "We matched the Cert Name Mismatch rule\n";
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a>: <b>Requested DNS name does not match the servers certificate!!</b>';
 push @html, '<p>Check the cert that the server is providing. If the cert appears fine, make sure the utility is using the name you are expecting.';
 push @html, 'Depending on the browser, a client might not notice this, but it is best practice to fix this issue.</p>';
 }
 #Matched SSL chain failures
 elsif ($stderr =~ /curl: \(60\)/) {
 
 $b = 1;
 
 # There was an SSL issue. Lets run our curl check again without forcing chain validation and continue
 undef $check;
 $check = `curl -k '$apps[$a]' -m 60 2> stderr.txt`;
 
 print "We matched the Cert-chain stderr log, we will run curl again in insecure mode\n";
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> <b style="color:green">is up</b> with caveats... <h3>NOTE:</h3>We found an issue with the certificate chain that your server provides. Validate the chain your server is sending.<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the chain. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 push @html, 'The application may be working, but best practice we should make sure the chain your server is sending is what it should.<br>';
 
 if ($sslv3check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 } 
 }
 
 # Comparison statement to see if the content of the curl contains validator
 
 # APP IS UP
 if (($check =~ /$validator[$a]/) && ($b == 0)) {
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> is <a style="color:green">up!!</a>';
 
 if (($sslv3check == 1) || ($sslv2check == 1)) {
 push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 } 
 if ($sslv3check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 }
 }
 # WE FOUND A 404
 elsif ($check =~ /404 - File or directory not found./) {
 
 $c = 1;
 
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> is showing a <b style="color:red">404 error</b>, check the server!!';
 }
 # WE FOUND A Server Error
 elsif ($check =~ /An unhandled exception occurred/) {
 
 $c = 1;
 
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> is showing a <b style="color:red">Server Error</b>, check the server!!';
 } 
 # DIDNT FIND VALIDATOR, RUN OTHER TESTS
 else {
 print "The first check failed, we will wait 30 seconds then check again.\n";
 # Wait 10 seconds, then run the test again, just to make sure
 `sleep 30s`;
 undef $check;
 $check = `curl '$apps[$a]' -m 60 2> stderr.txt`;
 
 # Skip this test if we matched something above
 if (($check !~ /$validator[$a]/) && ($b == 0)) { 
 $c = 1;
 
 push @html, '<hr><p>We didn\'t find this content: <i>';
 push @html, $validator[$a]; 
 push @html, '</i> in the <b>';
 push @html, $shortname[$a];
 push @html, '</b> application. <b style="color:red">The application appears to be down!!</p><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">Click here to check the URL</b></a><br>';
 
 # PORT CHECK SECTION
 # The content we expected was not found. As long as the host exists, lets run a port check on it. 
 if ($host ne "" ) {
 
 $ncoutput = `timeout 10s ncat -v '$host' '$port' &> ncat.tmp`;
 $cat = `cat ncat.tmp | grep Connected`;
 
 if ($cat =~ /Connected/) {
 push @html, '<h3>Summary:</h3><ul><li>From a network perspective, <b style="color:green">everything seems ok.</b></li>';
 push @html, '<li>We couldn\'t find the content we expected to see. We got there, but application may not be running/installed properly.</li>';
 push @html, '<li>If you see an \'Under Maintenance\' page, <b>check the inservice.txt file</b> on your server and make sure it loads locally.</li></ul>';
 push @html, 'Manually check the URL (above) and verify that what is presented is expected.</p>'; 
 
 if (($sslv3check == 1) || ($sslv2check == 1)) {
 push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 } 
 if ($sslv3check == 1) {
 push @html, '<b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 }
 }
 else {
 
 push @html, '<h3>Summary:</h3><ul><li>We couldnt find <b>content in the code</b> from the URL that we were expecting to see. </li>';
 push @html, '<li>We then ran a pinch test from the Internet and it <b>also failed.</b></li></ul>';
 push @html, 'Our guess is that the server is either <b>hard-down, DNS is not resolving</b> or something on the network is <b>not responding.</b><br>';
 push @html, 'Next step is to view the URL from the web. If nothing loads, check the same URL local to the server.<br>';
 push @html, 'If that loads you are most-likely facing a network/DNS issue.</b>';

 if (($sslv3check == 1) || ($sslv2check == 1)) {
 push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 } 
 if ($sslv3check == 1) {
 push @html, '<b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 }
 }
 }
 }
 # APP IS UP
 elsif (($check =~ /$validator[$a]/) && ($b == 0)) {
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> is <a style="color:green">up!!</a>';
 
 if (($sslv3check == 1) || ($sslv2check == 1)) {
 push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 } 
 if ($sslv3check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 }
 }
 }

 #######################################
 # CERT CHECK HTML #
 #######################################
 
 #If we couldn't pull a cert, prevent script from crashing
 if ($cert eq "") {
 print "We couldn't connect to the domain to run a cert check.";
 }
 else {
 # Convert year/mo/day to numeric date count
 $certnow = Date_to_Days(@certdates[0],@certdates[1],@certdates[2]);
 
 # Subtract current date and cert expiry date
 $certdaysleft = ($certnow - $now);
 
 #If the cert is beyond expiration, let us know
 if ($certdaysleft < 0) {
 push @html, '<br><b style="color:red">NOTE!!</b> The certificate for this domain has expired <b>' . $certdaysleft . "</b> days ago!!";
 }
 elsif ($certdaysleft <= 14) {
 print "\nCert expires in " . $certdaysleft . " days!!\n";
 push @html, '<br><b style="color:red">NOTE!!</b> The certificate for this domain will expire in <b>' . $certdaysleft . "</b> days!!";
 }
 }
 
 ####################################
 # ---- EMAIL CONFIGURATION ---- #
 ####################################

 if ($c == 1) {
 
 print "Compiling email...please wait...\n";
 
 $from = 'appchecker@mydomain.com';
 $to = 'me@mydomain.com';
 my $subject = $shortname[$a] . ' appears to be down!!';
 print $subject . "Sending notification email!!\n";
 my $body = '<font face="calibri">
<h2>Health Check Failed:</h2>

<a href="' . @apps[$a] . '" target="_blank">Click here to check URL</a>' . 
 '
<h4>Time of failure: </h4>

' . $time . 
 '

A public cloud server checks for content behind this URL periodically. We ran two tests against it, and couldn\'t find what we were looking for (' . $validator[$a] . 
 '). Click the URL to validate. After manually checking, if the page loads as you\'d expect, it may have been under heavy load at that given time. ' . 
 'Check historical monitoring tools for any possible outages or heavy load.

</font>';

 $msg = MIME::Lite->new(
 From => $from,
 To => $to,
 Subject => $subject,
 Data => qq{$body}
 );

 $msg->attr("content-type" => "text/html"); 
 $msg->send;
 }
 
 # Make sure our error parsing file is empty after each run
 `cat /dev/null > stderr.txt`;
 $a++;
 undef $b;
 undef $c;
 undef $check;
 
}


####################################
#  ---- SFTP Configuration ----    #
####################################
#
# Optionally upload HTML file to an SFTP site for viewing
# SFTP the file to the site of your choice
# Authentication method uses keys not interactive passwords

# Make sure index.html exists and is empty before we start
`touch index.html`;
`cat /dev/null > index.html`;

# Define filename
my $indexfile = 'index.html';

# Write HTML array to file
open (FILE, ">> $indexfile") || die "\nProblem opening $indexfile\n";
print FILE @html;
close (FILE);

# Create a simple batch file to put file on your SFTP site first
# Copy file to SFTP server
`sftp -b mysftpbatch.bat mysftpsite.domain.com`;

# Remove temporary file
`rm -f ncat.tmp`;

print "\nThe test is done, check the URL results\n";
Advertisements
Cloud-Based Web Application Check-Script

Deploy LAMP & phpMyAdmin on RHEL 6.5

lamp

I know there are a ton of articles out there on this topic. But I had a hard time finding one for RHEL 6.5 (NOT CentOS). There are subtle differences that lead to undoubted failure if you solely follow instructions meant for CentOS. So for all you RHEL people out there tasked with this, here you go.

It may or may-not work on different versions/distributions–no guarantee. I’m assuming you’ve installed the base image from ISO with no additional options and are waiting at the command line with privilege and Internet access.

Get your Repositories

Register your box with RHN.

# subscription-manager register --username user --password password

EPEL is required to properly install phpMyAdmin.

# wget http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
# rpm -ivh epel-release-6-8.noarch.rpm

The ‘optional’ repository is also needed to properly resolve dependencies.

# yum-config-manager --enable rhel-6-server-optional-rpms

Validate your repositories. Something like this should be displayed:

# yum repolist
repolist

Prep Steps

Update Server
# yum update -y
# reboot
Disable Firewall

I like to disable the local firewall, I use my hardware firewall for that. Modify iptables if the local firewall is something you want to do. There’s plenty on that out there.

# service iptables stop
# chkconfig iptables off
Install Apache
# yum install -y httpd
# service httpd start
# chkconfig httpd on
Validate Apache
# curl localhost | grep test

The HTML content will be visible from the test page here if everything went well.

Install PHP

Optionally install php-mysql if your application requires MySQL support with PHP.

# yum install -y php
# yum install -y php-mysql (optional)

Create a test PHP page.

# echo '<?php phpinfo() ; ?>' > /var/www/html/test.php

Validate the content loads. The command will display content based on your PHP config and versions.

# curl localhost/test.php

Remove the test page for security purposes

# rm -f /var/www/html/test.php
Install and start MySQL
# yum install -y mysql mysql-server
# service mysqld start
# chkconfig mysqld on

Secure and setup MySQL by running the vendor script. Press Enter at each prompt and set the root password when asked.

# mysql_secure_installation
Install phpMyAdmin (Optional)

PHPmyadmin is a GUI frontend for MySQL database management.

# yum install -y phpmyadmin

Configure remote access to phpmyadmin by editing the following file:

# vi /etc/httpd/conf.d/phpMyAdmin.conf

Do a search and replace for 127.0.0.1 and change to the IP you will use to manage MySQL through a browser, or add a subnet. In my case everything on my internal network starts with 10.x.x.x, so I did the following.

<Directory /usr/share/phpMyAdmin/>
   AddDefaultCharset UTF-8
   <IfModule mod_authz_core.c>
     # Apache 2.4
     <RequireAny>
       Require ip 10.0.0.0/8
       Require ip ::1
     </RequireAny>
   </IfModule>
   <IfModule !mod_authz_core.c>
     # Apache 2.2
     Order Deny,Allow
     Deny from All
     Allow from 10.0.0.0/8
     Allow from ::1
   </IfModule>
</Directory>
<Directory /usr/share/phpMyAdmin/setup/>
   <IfModule mod_authz_core.c>
     # Apache 2.4
     <RequireAny>
       Require ip 10.0.0.0/8
       Require ip ::1
     </RequireAny>
   </IfModule>
   <IfModule !mod_authz_core.c>
     # Apache 2.2
     Order Deny,Allow
     Deny from All
     Allow from 10.0.0.0/8
     Allow from ::1
   </IfModule>
</Directory>

Change authentication type to http.

# cp /usr/share/phpMyAdmin/config.sample.inc.php /usr/share/phpMyAdmin/config.inc.php 
# vi /usr/share/phpMyAdmin/config.inc.php

Change cookie to http

[...] 
/* Authentication type */
$cfg['Servers'][$i]['auth_type'] = 'http';
[...]

Finish Up

I like to reboot for good measure, making sure your services start on their own and are functional.

# reboot

Finally, open a browser on the machine you allowed in the steps above and browse to the following:

http://mylampserver/phpmyadmin 

Log in with the root user you created for MySQL.

That’s it, enjoy!!

Deploy LAMP & phpMyAdmin on RHEL 6.5

Create a Software Load Balancer w/ Content Switching and SSL

I was posed with a problem at work. We have a hardware load-balancer in our production environment (Cisco ACE). We were required to build out a disaster recovery site without the funds to purchase a hardware load-balancer. I then began my research to make it happen. I had a really hard time finding a document for this that was end-to-end, so I decided to write this.

This was originally created back in 2011 so some of the software versions may be a little behind. You can try newer versions if you like, the process should be fairly similar.

The load-balancer software is called HAproxy and it runs on Linux systems. You can install it on any distribution that you like, but for this guide I used Ubuntu Server 9.10. I cant guarantee the same results from a different distro.

Install the OS

  1. Download Ubuntu server 9.10 and install it as a VM or a physical server.
  2. Make sure you have correct IP settings so the server can access the internet, or else apt will not be able to install properly.
  3. When the software package installation screen comes up, only install OpenSSH server.
  4. Log in as the user you specified at installation.
  5. Enter the su -i command to gain privileges to reset the root password
  6. Enter passwd root to change the password to something specific.

Install HAProxy

  1. Once you get your OS up and running you need to install HAproxy.
  2. Install HAproxy with apt. Run the following command (you have to be logged in as root or use su -i)
  • apt-get install haproxy
  1. Back up the old config file
    • cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxyORIGINAL.cfg
  2. Paste the following into your config file and build off of it (this contains content switching)
global
   log 127.0.0.1 local0
   maxconn 4096
   uid 99
   gid 99
   daemon

defaults
   mode http
   log global
   option tcplog
   option httpclose
   retries 3
   maxconn 2000
   contimeout      5000
   clitimeout      50000
   srvtimeout      50000

frontend LB1 *:80
   option forwardfor
   reqadd          X-Forwarded-Proto:\ https
   reqadd          FRONT_END_HTTPS:\ on
   acl FARM1-acl url_sub -i Hello
   acl FARM2-acl url_sub -i Goodbye
   use_backend Hello if FARM1-acl
   use_backend Goodbye if FARM2-acl
   default_backend DEFAULT

backend FARM1
   stats enable
   stats auth admin:password10
   balance roundrobin
   #option ssl-hello-chk
   server SERVER1 10.0.5.1:80 check
   server SERVER2 10.0.5.2:80 check

backend FARM2
   stats enable
   stats auth admin:password10
   balance roundrobin
   #option ssl-hello-chk
   server SERVER3 10.0.5.3:80 check
   server SERVER4 10.0.5.4:80 check

backend DEFAULT
   stats enable
   stats auth admin:password10
   balance roundrobin
   #option ssl-hello-chk
   server SERVER5 10.0.5.5:80 check
   server SERVER6 10.0.5.6:80 check
  1. Now set HAProxy to start on reboot.
    • vim /etc/default/haproxy
    • Change the ENABLED = 0 to 1
  2. Restart the service and make sure you can hit your frontend IP on port 80.
    • service haproxy restart
  3. When you make changes to the /etc/haproxy.cfg file make sure to run service haproxy restart


Install Stunnel

Stunnel handles the SSL backend connections. Without Stunnel, content switching would not be possible as the traffic would be encrypted.
  1. Create a directory to download stunnel and it’s patch to
    • mkdir /root/stunnel
    • cd /root/stunnel
  2. Download Stunnel and it’s patch (in this version i used version 4.32)
  3. Patch Stunnel
    • cd stunnel-4.32
    • patch -p1 < ../stunnel-4.32-xforwarded-for.diff
  4. Install prerequisites
    • apt-get install libcurl3-openssl-dev
  5. Install stunnel
    • ./configure
    • make && make install

Generate Self-signed Certs

This step is for TESTING ONLY. Do this if if you dont have your real certs yet. To verify everything you’ll need a certificate and private key.
  1. Install OpenSSL:
    • apt-get install openssl
  2. Make a certificate directory:
    • mkdir /etc/certs
    • cd /etc/certs
  3. Generate Private key:
    • openssl genrsa -out server.key 1024
  4. Generate CSR:
    • openssl req -new -key server.key -out server.csr
  5. Generate Self-signed cert
    • openssl x509 -req -days 1200 -in server.csr -signkey server.key -out server.crt
  6. Make the stunnel config file
    • mkdir /etc/stunnel
    • vim /etc/stunnel/stunnel.conf
  7. Add the following to the basic config for stunnel:
cert=/etc/certs/server.crt
key = /etc/certs/server.key
;setuid = nobody
;setgid = nogroup

pid = /etc/stunnel/stunnel.pid
debug = 3
output = /etc/stunnel/stunnel.log

socket=l:TCP_NODELAY=1
socket=r:TCP_NODELAY=1

[https]
accept=192.168.1.1:443
connect=192.168.1.1:80
TIMEOUTclose=0
xforwardedfor=yes
  1. Set stunnel to start on reboot
    • vim /etc/init.d/stunnel
    • Paste the following into that file:
#!/bin/bash
#
# stunnel      This shell script takes care of starting and stopping
#              stunnel
#
# chkconfig: 345 80 30
# description:  Secure tunnel

# processname: stunnel
# config: /etc/stunnel/stunnel.conf
# pidfile: /var/run/stunnel/stunnel.pid

# Source function library.
. /lib/lsb/init-functions

# Source stunnel configureation.
if [ -f /etc/sysconfig/stunnel ] ; then
 . /etc/sysconfig/stunnel
fi

RETVAL=0
prog="stunnel"

start() {
 # Start daemons.

 echo -n $"Starting $prog: "
 if test -x /usr/local/bin/stunnel ; then
   /usr/local/bin/stunnel /etc/stunnel/stunnel.conf
 fi
 RETVAL=$?
 echo
 [ $RETVAL -eq 0 ] && touch /var/lock/stunnel
 return $RETVAL
}

stop() {
 # Stop daemons.
 echo -n $"Shutting down $prog: "
 killproc stunnel
 RETVAL=$?
 echo
 [ $RETVAL -eq 0 ] && rm -f /var/lock/stunnel
 return $RETVAL
}

# See how we were called.
case "$1" in
  start)
 start
 ;;
  stop)
 stop
 ;;
  restart)
 stop
 start
 RETVAL=$?
 ;;
  condrestart)
 if [ -f /var/lock/stunnel ]; then
     stop
     start
     RETVAL=$?
 fi
 ;;
  status)
 status stunnel
 RETVAL=$?
 ;;
  *)
 echo $"Usage: $0 {start|stop|restart|condrestart|status}"
 exit 1
esac

exit $RETVAL
  1. Save it and set it so it can be executed
    • chmod 755 /etc/init.d/stunnel
  2. Test the service
    • Service stunnel stop
    • Service stunnel start
    • (you might need to run the start command a couple times before it works)
  3. Install chkconfig so the service can start on it’s own
    • apt-get install chkconfig
    • chkconfig stunnel on
  4. Restart the server for good measure and make sure your services start.
  5. Finally test the function of HAproxy by going to http://host and https://host with the respective URL paths to test content switching.

Export Real Certificate

When you are done testing you will want to use the actual certificate for the servers that you are load-balancing for. In my case the real certificate was on a windows box. I had to export the certificate and private key from IIS and import it into something stunnel could understand.
  1. Go to the IIS server that has the certificate.
  2. Browse to the site that has the cert and right click it then go to properties.
  3. Click the Directory Security tab and click View Certificate…
  4. Click Details → Copy to File…
  5. Click Yes, export the private key
  6. Uncheck enable strong protection and click next. Make sure all sub boxes are not checked.
  7. Leave the password fields blank
  8. Pick the path where you want to save it.
  9. Click Next → Finish
  10. Take that file and copy it to your load balancer
  11. Once it is copied over cd to the directory that you moved it to and run the following command:
    • openssl pkcs12 -in mycertificate.pfx -nodes -out CERT.pem
  12. Generate your private key associated with the cert.
    • openssl rsa CERT.pem -out mycertificate.key
  13. Edit your /etc/stunnel.conf file to reflect the changes. E.g.
    • cert=/etc/certs/CERT.pem
  14. Restart stunnel

Enable Logging

  1. Install syslog if it is not already installed
    • apt-get install syslogd
  2. Change the config of syslog to allow socket connections
    • vim /etc/default/syslogd
    • SYSLOGD=”-r”
  3. Set the path to log to
    • vim /etc/syslog.conf
    • local0.* /var/log/messages
  4. Restart syslog
    • /etc/init.d/sysklogd restart
  5. Make sure haproxy is set up to log
global
  log 127.0.0.1 local0

defaults
  log global
  option tcplog

 For Servers that have Backend SSL’s

Use nginx. The load-balancing algorithm isnt as effictive as HAProxy, but HAProxy does not support servers on the backend with SSL certificates. This example uses the same base version of ubuntu server with SSH server installed only.
  1. Install nginx
    • apt-get install nginx
      OR
    • Install from source:
./configure  --conf-path=/etc/nginx/nginx.conf --with-http_dav_module  --with-http_ssl_module  --with-http_stub_status_module  --add-module=/root/http-healthcheck  --add-module=/root/upstream

aptitude show nginx
apt-get install libc6
apt-get install libgd2-noxpm
apt-get install libgd2-xpm libgeoip1 libpcre3 libssl0.9.8 libxml2 libxslt1.1 zlib1g lsb-base
  1. Install chkconfig
    • apt-get install chkconfig
  2. See if nginx starts with the default config
    • service nginx restart
  3. Make sure you have your SSL certificate and key ready. Without it the next steps arent going to work. Refer the above section on “Export Real SSL
  4. Modify the default nginx config. This example uses content filtering with front-end SSL proxying.
user www-data;
worker_processes  2; #this server has two processors
worker_cpu_affinity 01 10; #assigning each worker to a processor

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;

    access_log  /var/log/nginx/access.log;

    sendfile        on;

    keepalive_timeout  65;
    tcp_nodelay        on;

    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;

#defining the farms and backend ports
upstream DEFAULT {
  server 10.0.5.5:443;
  server 10.0.5.6:443;
 }

upstream FARM1 {
  server 10.0.5.1:443;
  server 10.0.5.2:443;
}

upstream FARM2 {
  server 10.0.5.3:443;
  server 10.0.5.4:443;
}

#define the server
server {

  listen 10.0.5.50:443;

  ssl on;
  ssl_certificate /etc/certs/CERT.pem; #location of cert
  ssl_certificate_key /etc/certs/KEY.key; #location of key

  server_name host.header.com;

#location is where content filtering happens. Each path after the / will get directed to the 
#farm defined below w/ the proxy_pass command
  location ~* ^/virtual1|virtual2|virtual3 {

   proxy_set_header  X-Real-IP  $remote_addr;

   proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header Host $http_host;
   proxy_redirect off;
   proxy_max_temp_file_size 0;

   proxy_pass https://VIRTUAL1;

  }

  location ~* ^/virtual4|virtual5 {

   proxy_set_header  X-Real-IP  $remote_addr;

   proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header Host $http_host;
   proxy_redirect off;
   proxy_max_temp_file_size 0;

   proxy_pass https://VIRTUAL5;

  }

  location ~* ^/virtual6|virtual7 {

   proxy_set_header  X-Real-IP  $remote_addr;

   proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header Host $http_host;
   proxy_redirect off;
   proxy_max_temp_file_size 0;

   proxy_pass https://VIRTUAL6;

  }

 }

}
  1. Restart nginx
    • service nginx restart
  2. Set nginx to start on reboot
    • chkconfig nginx on
  3. Make sure nginx is running properly with the netstat -lp command
 
root@LBHOST:~# netstat -lp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 LBHOST.local:https *:*                     LISTEN      1322/nginx
tcp        0      0 *:www                   *:*                     LISTEN      1322/nginx
tcp        0      0 *:ssh                   *:*                     LISTEN      882/sshd
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN      882/sshd
Create a Software Load Balancer w/ Content Switching and SSL