Cloud-Based Web Application Check-Script

production_inspection

Monitoring applications can be tricky. As a network engineer it’s important that the applications that I serve up are available and working properly. To network engineers it’s important that resources are up and running from the Internet. This script checks a variety of aspects pertaining to a public URL.

  • Availability (TCP ports)
  • Content matching (HTTP GET)
  • Certificate Expiration
  • Security (SSL disabled, proper chain, etc.)
  • DNS (cert name mismatch, DNS lookup)

The script runs from a Linux server I have in the cloud. Use any platform you choose. All that is required is Perl, curl, openssl and netcat.The MIME::Lite and DATE::Calc Perl modules are also required.

The script will run through a list of URL’s defined by you and check for content we’d expect when the application is up. We run through a lot of other tests mentioned above. Take a look at the script and see what it checks for.

The script will push output to an HTML file that allows for “user-friendly” appearance in email alerts and the HTML file can be uploaded to a webserver.

Configure the script to run periodically with a cron job.

This is a free, easily modified script that can help monitor your applications without much overhead. Use and modify to your specific needs.

#!/usr/bin/perl
use MIME::Lite;
use Date::Calc qw(
Date_to_Days);

#############################################################
# -----VERSION NOTES-----              
# 1.0: Base configuration
# 1.1: Added color/formatting changes to output
# 1.2: Added ncat port validation after failure
# 1.3: Added email functionality
# 1.4: Modified code to send HTML instead of text
# 1.5: Save HTML output to file and SFTP to dashboard server
# 1.6: Added additional logic for failure types, e.g. 404, etc.
# 2.0: Added Email logic for detecting failures
# 2.1: Added cert checking logic


#############################################################
# To test an application, add the full URL of the app you
# want to test to the apps variable. Secondly add a string of 
# content that matches when the application is up and running. 
# If the content doesn't match, the application is marked as 
# down. 
#############################################################

# Applications short or common name goes here e.g. 'Customer WebUI'
@shortname = (	#App1
				'Google',
				#App2
				'Fake Domain'
			);
			  
# FULL application URL goes here to test for availability and content
# !!!!!MAKE SURE YOU ADD THE PORT NUMBER AFTER THE HOSTNAME FOR THE TEST TO WORK!!!!!
@apps = (	'https://www.google.com:443/',
			'https://fakedomain.domains.net:443/'
		);

# Strings to match against when you know the application is up go here. 
@validator = (	'Google Search',
				'Bad'
			);
			  
# Count number of variables in array to loop can run properly
$numapps = (scalar @apps);

# Define variables

# Needed for our while statement
$a = 0;
# If one failure check passed, don't check for other failures. Reduces redundant information.
$b = 0;
# If c = 1, app is down. Used for e-mail notifications
$c = 0;

# Define HTML array to be sent via. SFTP
my @html = ();

# Set datetime
$time = `TZ=":US/Eastern" date`;

# Add environmental header in HTML file
push @html, '<head><title>App Status</title></head>';
push @html, '

<h1>Application Status</h1>


';
push @html, '

<h3>The applications listed here are checked once every 30 minutes</h3>


';
push @html, '

Last Checked: ' ;
push @html, $time;
push @html, '

';

# Tell the user that something is happening
print "The test is running, please wait... \n";

# Begin the checks
while($a < $numapps){

 # Grab URL components to allow for more segmented testing, use variables as needed
 my($protocol, $host, $port, $uri) = $apps[$a] =~ m|(.*)://([a-zA-Z0-9\-.]+):([0-9]+)?(.*)?|;

 # Lets run some quick security tests first
 $sslv2check = `timeout 3 openssl s_client -connect '$host':443 -ssl2 2>/dev/null`;
 $sslv3check = `timeout 3 openssl s_client -connect '$host':443 -ssl3 2>/dev/null`;

 #********************
 #* CERT DATE LOGIC *
 #********************
 
 # Get todays year, month and date
 $year = `TZ=":UTC" date +%Y`;
 $month = `TZ=":UTC" date +%m`;
 $date = `TZ=":UTC" date +%d`;
 
 #Convert todays date to a numeric date count
 $now = Date_to_Days($year,$month,$date);
 
 # Grab certificate date
 $cert = `timeout 3 openssl s_client -connect $host:443 -tls1 2>/dev/null| openssl x509 -noout -enddate 2>/dev/null| cut -f2 -d'=' | xargs -0 date +%F -d 2>/dev/null`;

 # Split the output into DATE MONTH DAY
 my @certdates = split('-', $cert); 


 #End certificate expiration logic
 
 if ($sslv3check =~ /Server public key is/) {
 $sslv3check = 1;
 print "We matched the SSLv3 security check\n";
 }
 if ($sslv2check =~ /Server public key is/) {
 $sslv2check = 1;
 print "We matched the SSLv2 security check\n";
 }

 # The standard check with a 60 second timeout. This is what grabs the HTTP information response from server
 $check = `curl '$apps[$a]' -m 60 2> stderr.txt`;
 
 # If the check failed, we want to preserve that information. Stderr is saved to a variable and the checks below are ran. 
 undef $stderr;
 my $stderr = `cat stderr.txt`;
 
 # Matched DNS unresolvable 
 if ($stderr =~ /curl: \(6{1}\)/) {
 
 # I set this here so we don't run other redundant checks.
 $b = 1;
 
 print "We matched the DNS stderr log\n";
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a>: <b>We couldn\'t resolve the host that you specified</b> using public DNS servers. Something is amiss...'; 
 }
 #Matched a TCP reset
 elsif ($stderr =~ /curl: \(56\)/) {
 
 $b = 1;
 
 print "We matched the TCP Reset rule\n";
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a>: <b>We were sent a reset!!</b> Since we did not see the construction page, there are a few reasons for this:<ul>';
 push @html, '<li>The server is not accepting requests at all. Check the server locally and make sure it does the same with localhost.</li>';
 push @html, '<li>The firewall is blocking us, DNS could be wrong, or the firewall isn\'t listening on this port</li></ul>'; 
 } 
 #Matched a cert name mismatch
 elsif ($stderr =~ /curl: \(51\)/) {
 
 undef $check;
 $check = `curl -k '$apps[$a]' -m 60 2> stderr.txt`;
 
 print "We matched the Cert Name Mismatch rule\n";
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a>: <b>Requested DNS name does not match the servers certificate!!</b>';
 push @html, '<p>Check the cert that the server is providing. If the cert appears fine, make sure the utility is using the name you are expecting.';
 push @html, 'Depending on the browser, a client might not notice this, but it is best practice to fix this issue.</p>';
 }
 #Matched SSL chain failures
 elsif ($stderr =~ /curl: \(60\)/) {
 
 $b = 1;
 
 # There was an SSL issue. Lets run our curl check again without forcing chain validation and continue
 undef $check;
 $check = `curl -k '$apps[$a]' -m 60 2> stderr.txt`;
 
 print "We matched the Cert-chain stderr log, we will run curl again in insecure mode\n";
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> <b style="color:green">is up</b> with caveats... <h3>NOTE:</h3>We found an issue with the certificate chain that your server provides. Validate the chain your server is sending.<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the chain. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 push @html, 'The application may be working, but best practice we should make sure the chain your server is sending is what it should.<br>';
 
 if ($sslv3check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 } 
 }
 
 # Comparison statement to see if the content of the curl contains validator
 
 # APP IS UP
 if (($check =~ /$validator[$a]/) && ($b == 0)) {
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> is <a style="color:green">up!!</a>';
 
 if (($sslv3check == 1) || ($sslv2check == 1)) {
 push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 } 
 if ($sslv3check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 }
 }
 # WE FOUND A 404
 elsif ($check =~ /404 - File or directory not found./) {
 
 $c = 1;
 
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> is showing a <b style="color:red">404 error</b>, check the server!!';
 }
 # WE FOUND A Server Error
 elsif ($check =~ /An unhandled exception occurred/) {
 
 $c = 1;
 
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> is showing a <b style="color:red">Server Error</b>, check the server!!';
 } 
 # DIDNT FIND VALIDATOR, RUN OTHER TESTS
 else {
 print "The first check failed, we will wait 30 seconds then check again.\n";
 # Wait 10 seconds, then run the test again, just to make sure
 `sleep 30s`;
 undef $check;
 $check = `curl '$apps[$a]' -m 60 2> stderr.txt`;
 
 # Skip this test if we matched something above
 if (($check !~ /$validator[$a]/) && ($b == 0)) { 
 $c = 1;
 
 push @html, '<hr><p>We didn\'t find this content: <i>';
 push @html, $validator[$a]; 
 push @html, '</i> in the <b>';
 push @html, $shortname[$a];
 push @html, '</b> application. <b style="color:red">The application appears to be down!!</p><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">Click here to check the URL</b></a><br>';
 
 # PORT CHECK SECTION
 # The content we expected was not found. As long as the host exists, lets run a port check on it. 
 if ($host ne "" ) {
 
 $ncoutput = `timeout 10s ncat -v '$host' '$port' &> ncat.tmp`;
 $cat = `cat ncat.tmp | grep Connected`;
 
 if ($cat =~ /Connected/) {
 push @html, '<h3>Summary:</h3><ul><li>From a network perspective, <b style="color:green">everything seems ok.</b></li>';
 push @html, '<li>We couldn\'t find the content we expected to see. We got there, but application may not be running/installed properly.</li>';
 push @html, '<li>If you see an \'Under Maintenance\' page, <b>check the inservice.txt file</b> on your server and make sure it loads locally.</li></ul>';
 push @html, 'Manually check the URL (above) and verify that what is presented is expected.</p>'; 
 
 if (($sslv3check == 1) || ($sslv2check == 1)) {
 push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 } 
 if ($sslv3check == 1) {
 push @html, '<b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 }
 }
 else {
 
 push @html, '<h3>Summary:</h3><ul><li>We couldnt find <b>content in the code</b> from the URL that we were expecting to see. </li>';
 push @html, '<li>We then ran a pinch test from the Internet and it <b>also failed.</b></li></ul>';
 push @html, 'Our guess is that the server is either <b>hard-down, DNS is not resolving</b> or something on the network is <b>not responding.</b><br>';
 push @html, 'Next step is to view the URL from the web. If nothing loads, check the same URL local to the server.<br>';
 push @html, 'If that loads you are most-likely facing a network/DNS issue.</b>';

 if (($sslv3check == 1) || ($sslv2check == 1)) {
 push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 } 
 if ($sslv3check == 1) {
 push @html, '<b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 }
 }
 }
 }
 # APP IS UP
 elsif (($check =~ /$validator[$a]/) && ($b == 0)) {
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> is <a style="color:green">up!!</a>';
 
 if (($sslv3check == 1) || ($sslv2check == 1)) {
 push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 } 
 if ($sslv3check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 }
 }
 }

 #######################################
 # CERT CHECK HTML #
 #######################################
 
 #If we couldn't pull a cert, prevent script from crashing
 if ($cert eq "") {
 print "We couldn't connect to the domain to run a cert check.";
 }
 else {
 # Convert year/mo/day to numeric date count
 $certnow = Date_to_Days(@certdates[0],@certdates[1],@certdates[2]);
 
 # Subtract current date and cert expiry date
 $certdaysleft = ($certnow - $now);
 
 #If the cert is beyond expiration, let us know
 if ($certdaysleft < 0) {
 push @html, '<br><b style="color:red">NOTE!!</b> The certificate for this domain has expired <b>' . $certdaysleft . "</b> days ago!!";
 }
 elsif ($certdaysleft <= 14) {
 print "\nCert expires in " . $certdaysleft . " days!!\n";
 push @html, '<br><b style="color:red">NOTE!!</b> The certificate for this domain will expire in <b>' . $certdaysleft . "</b> days!!";
 }
 }
 
 ####################################
 # ---- EMAIL CONFIGURATION ---- #
 ####################################

 if ($c == 1) {
 
 print "Compiling email...please wait...\n";
 
 $from = 'appchecker@mydomain.com';
 $to = 'me@mydomain.com';
 my $subject = $shortname[$a] . ' appears to be down!!';
 print $subject . "Sending notification email!!\n";
 my $body = '<font face="calibri">
<h2>Health Check Failed:</h2>

<a href="' . @apps[$a] . '" target="_blank">Click here to check URL</a>' . 
 '
<h4>Time of failure: </h4>

' . $time . 
 '

A public cloud server checks for content behind this URL periodically. We ran two tests against it, and couldn\'t find what we were looking for (' . $validator[$a] . 
 '). Click the URL to validate. After manually checking, if the page loads as you\'d expect, it may have been under heavy load at that given time. ' . 
 'Check historical monitoring tools for any possible outages or heavy load.

</font>';

 $msg = MIME::Lite->new(
 From => $from,
 To => $to,
 Subject => $subject,
 Data => qq{$body}
 );

 $msg->attr("content-type" => "text/html"); 
 $msg->send;
 }
 
 # Make sure our error parsing file is empty after each run
 `cat /dev/null > stderr.txt`;
 $a++;
 undef $b;
 undef $c;
 undef $check;
 
}


####################################
#  ---- SFTP Configuration ----    #
####################################
#
# Optionally upload HTML file to an SFTP site for viewing
# SFTP the file to the site of your choice
# Authentication method uses keys not interactive passwords

# Make sure index.html exists and is empty before we start
`touch index.html`;
`cat /dev/null > index.html`;

# Define filename
my $indexfile = 'index.html';

# Write HTML array to file
open (FILE, ">> $indexfile") || die "\nProblem opening $indexfile\n";
print FILE @html;
close (FILE);

# Create a simple batch file to put file on your SFTP site first
# Copy file to SFTP server
`sftp -b mysftpbatch.bat mysftpsite.domain.com`;

# Remove temporary file
`rm -f ncat.tmp`;

print "\nThe test is done, check the URL results\n";
Advertisements
Cloud-Based Web Application Check-Script

How to create an SSL Monitoring Server

If you have SSL certificates on many servers across the network it can sometimes be difficult to manage expiration dates for all the certs. There is a UNIX utility called “ssl-cert-check” that can scan servers on any port and notify you if they are near expiration.
This how-to is based on Cygwin instead of a *nix distro. The steps should be the same, but there may be some differences.
1. Install Cygwin (skip to step 3 if you are running a Linux host)
2. Download from the Cygwin website and make sure OpenSSL, mail, cygrunsrv, cron and vim are installed.
3. Once Cygwin is installed you need to download the ssl-check software
  $ wget http://prefetch.net/code/ssl-cert-check
  $ chmod 775 ssl-cert-check
  $ mv ssl-cert-check /bin
4. Test that the tool is working
  $ cd /bin
  $ ssl-cert-check -s x.x.x.x -p 443
5. Once you know the tool is working you can start customizing it to fit your needs. I needed to scan a large group of servers on port 443 and email me if any of them were within 14 days of expiring.
6. I created a text file with the hosts that I needed to scan and the ports that certificates were bound to.
  $ vi ssl-servers

  www.server.com 443
  192.168.0.22 443
  example.com 443
8. Once you have the file you can use ssl-cert-check to scan using the file. Test to make sure the file is working.
$ ssl-cert-check -i -f ssl-servers
Host                                Issuer            Status   Expires     Days Left
----------------------------------- ----------------- -------- ----------- ---------
www.server.com:443                  Comodo Limited    Valid    May 23 2012 218
192.168.0.22:443                    Equifax Secure In Valid    Jun 20 2012 246
example.com:443                     Thawte Consulting Valid    Jun 7 2012  233
9. Now that I know that I can scan against a list of servers, I want to be notified by email.
10. I want to test to make sure that email is installed and that my host can send out an email properly.
$ cd /usr/bin
$ mail -f MYSERVER@domain.com -s Test -r MAILSERVER user.name@domain.com
11. In order for Cygwin to be able to automatically send emails you need to modify a few things. First create a symbolic link pointing to the proper mail binary.
$ ln -s /bin/mail /usr/lib/sendmail
12. Then you need to modify the mail config file. Make sure the proper SMTP server and port is uncommented. Then add a name and email to send FROM.
$ vi /etc/email/email.conf
############################################################
# SMTP Server and Port number you use
############################################################
 SMTP_SERVER = '192.168.0.50'
 SMTP_PORT = '25'

############################################################
# If you'd rather use sendmail binary, specify it and the
# command line switches to use, here.  If you have both
# this option and SMTP_SERVER set, SMTP_SERVER will be of
# higher priority than SENDMAIL.
############################################################
SENDMAIL_BIN = '/usr/lib/sendmail -t -i'

############################################################
# Your email address: If you'd like To have your name to
# show in the from field instead of just your email address,
# then keep the format below and edit it to your email
# and name.
############################################################
MY_NAME  = 'SSL Monitoring Tool'
MY_EMAIL = 'server@domain.com'
13. Once your email configuration is finished you can test the following command which will check against a list of servers and email to the email you specify if the cert is due to expire in 60 days.
$ ssl-cert-check -a -f ssl-servers -q -x 60 -e user@domain.com
14. Install cron as a service, first right-click the Cygwin icon on the desktop and select “Run as Administrator”
$ cron-config
$ yes|ntsec smbntsec|no|enter password
15. Once cron is installed you can modify the cron tasks by running crontab -e at the command prompt
NOTE: If it’s a Windows 2008 server, chances are you might have trouble installing the cron service. If this is the case you can run any Cygwin task from the built-in Windows Scheduler service. Just save your command as a batch file and schedule it. E.g.
C:\Cygwin\bin\bash -c "./ssl-cert-check -a -q -f /bin/ssl-servers -x 14 -e userk@domain.com"
How to create an SSL Monitoring Server