Monitoring applications can be tricky. As a network engineer it’s important that the applications that I serve up are available and working properly. To network engineers it’s important that resources are up and running from the Internet. This script checks a variety of aspects pertaining to a public URL.
- Availability (TCP ports)
- Content matching (HTTP GET)
- Certificate Expiration
- Security (SSL disabled, proper chain, etc.)
- DNS (cert name mismatch, DNS lookup)
The script runs from a Linux server I have in the cloud. Use any platform you choose. All that is required is Perl, curl, openssl and netcat.The MIME::Lite and DATE::Calc Perl modules are also required.
The script will run through a list of URL’s defined by you and check for content we’d expect when the application is up. We run through a lot of other tests mentioned above. Take a look at the script and see what it checks for.
The script will push output to an HTML file that allows for “user-friendly” appearance in email alerts and the HTML file can be uploaded to a webserver.
Configure the script to run periodically with a cron job.
This is a free, easily modified script that can help monitor your applications without much overhead. Use and modify to your specific needs.
#!/usr/bin/perl use MIME::Lite; use Date::Calc qw( Date_to_Days); ############################################################# # -----VERSION NOTES----- # 1.0: Base configuration # 1.1: Added color/formatting changes to output # 1.2: Added ncat port validation after failure # 1.3: Added email functionality # 1.4: Modified code to send HTML instead of text # 1.5: Save HTML output to file and SFTP to dashboard server # 1.6: Added additional logic for failure types, e.g. 404, etc. # 2.0: Added Email logic for detecting failures # 2.1: Added cert checking logic ############################################################# # To test an application, add the full URL of the app you # want to test to the apps variable. Secondly add a string of # content that matches when the application is up and running. # If the content doesn't match, the application is marked as # down. ############################################################# # Applications short or common name goes here e.g. 'Customer WebUI' @shortname = ( #App1 'Google', #App2 'Fake Domain' ); # FULL application URL goes here to test for availability and content # !!!!!MAKE SURE YOU ADD THE PORT NUMBER AFTER THE HOSTNAME FOR THE TEST TO WORK!!!!! @apps = ( 'https://www.google.com:443/', 'https://fakedomain.domains.net:443/' ); # Strings to match against when you know the application is up go here. @validator = ( 'Google Search', 'Bad' ); # Count number of variables in array to loop can run properly $numapps = (scalar @apps); # Define variables # Needed for our while statement $a = 0; # If one failure check passed, don't check for other failures. Reduces redundant information. $b = 0; # If c = 1, app is down. Used for e-mail notifications $c = 0; # Define HTML array to be sent via. SFTP my @html = (); # Set datetime $time = `TZ=":US/Eastern" date`; # Add environmental header in HTML file push @html, '<head><title>App Status</title></head>'; push @html, ' <h1>Application Status</h1> '; push @html, ' <h3>The applications listed here are checked once every 30 minutes</h3> '; push @html, ' Last Checked: ' ; push @html, $time; push @html, ' '; # Tell the user that something is happening print "The test is running, please wait... \n"; # Begin the checks while($a < $numapps){ # Grab URL components to allow for more segmented testing, use variables as needed my($protocol, $host, $port, $uri) = $apps[$a] =~ m|(.*)://([a-zA-Z0-9\-.]+):([0-9]+)?(.*)?|; # Lets run some quick security tests first $sslv2check = `timeout 3 openssl s_client -connect '$host':443 -ssl2 2>/dev/null`; $sslv3check = `timeout 3 openssl s_client -connect '$host':443 -ssl3 2>/dev/null`; #******************** #* CERT DATE LOGIC * #******************** # Get todays year, month and date $year = `TZ=":UTC" date +%Y`; $month = `TZ=":UTC" date +%m`; $date = `TZ=":UTC" date +%d`; #Convert todays date to a numeric date count $now = Date_to_Days($year,$month,$date); # Grab certificate date $cert = `timeout 3 openssl s_client -connect $host:443 -tls1 2>/dev/null| openssl x509 -noout -enddate 2>/dev/null| cut -f2 -d'=' | xargs -0 date +%F -d 2>/dev/null`; # Split the output into DATE MONTH DAY my @certdates = split('-', $cert); #End certificate expiration logic if ($sslv3check =~ /Server public key is/) { $sslv3check = 1; print "We matched the SSLv3 security check\n"; } if ($sslv2check =~ /Server public key is/) { $sslv2check = 1; print "We matched the SSLv2 security check\n"; } # The standard check with a 60 second timeout. This is what grabs the HTTP information response from server $check = `curl '$apps[$a]' -m 60 2> stderr.txt`; # If the check failed, we want to preserve that information. Stderr is saved to a variable and the checks below are ran. undef $stderr; my $stderr = `cat stderr.txt`; # Matched DNS unresolvable if ($stderr =~ /curl: \(6{1}\)/) { # I set this here so we don't run other redundant checks. $b = 1; print "We matched the DNS stderr log\n"; push @html, '<hr><a href="'; push @html, @apps[$a]; push @html, '" target="_blank">'; push @html, $shortname[$a]; push @html, '</a>: <b>We couldn\'t resolve the host that you specified</b> using public DNS servers. Something is amiss...'; } #Matched a TCP reset elsif ($stderr =~ /curl: \(56\)/) { $b = 1; print "We matched the TCP Reset rule\n"; push @html, '<hr><a href="'; push @html, @apps[$a]; push @html, '" target="_blank">'; push @html, $shortname[$a]; push @html, '</a>: <b>We were sent a reset!!</b> Since we did not see the construction page, there are a few reasons for this:<ul>'; push @html, '<li>The server is not accepting requests at all. Check the server locally and make sure it does the same with localhost.</li>'; push @html, '<li>The firewall is blocking us, DNS could be wrong, or the firewall isn\'t listening on this port</li></ul>'; } #Matched a cert name mismatch elsif ($stderr =~ /curl: \(51\)/) { undef $check; $check = `curl -k '$apps[$a]' -m 60 2> stderr.txt`; print "We matched the Cert Name Mismatch rule\n"; push @html, '<hr><a href="'; push @html, @apps[$a]; push @html, '" target="_blank">'; push @html, $shortname[$a]; push @html, '</a>: <b>Requested DNS name does not match the servers certificate!!</b>'; push @html, '<p>Check the cert that the server is providing. If the cert appears fine, make sure the utility is using the name you are expecting.'; push @html, 'Depending on the browser, a client might not notice this, but it is best practice to fix this issue.</p>'; } #Matched SSL chain failures elsif ($stderr =~ /curl: \(60\)/) { $b = 1; # There was an SSL issue. Lets run our curl check again without forcing chain validation and continue undef $check; $check = `curl -k '$apps[$a]' -m 60 2> stderr.txt`; print "We matched the Cert-chain stderr log, we will run curl again in insecure mode\n"; push @html, '<hr><a href="'; push @html, @apps[$a]; push @html, '" target="_blank">'; push @html, $shortname[$a]; push @html, '</a> <b style="color:green">is up</b> with caveats... <h3>NOTE:</h3>We found an issue with the certificate chain that your server provides. Validate the chain your server is sending.<br>'; push @html, 'You can use <i>ssl-labs</i> to check the chain. Here is an automated link to check yourself.<ul><li>'; push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d='; push @html, $host; push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>'; push @html, 'The application may be working, but best practice we should make sure the chain your server is sending is what it should.<br>'; if ($sslv3check == 1) { push @html, '<br><b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>'; } if ($sslv2check == 1) { push @html, '<br><b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; } } # Comparison statement to see if the content of the curl contains validator # APP IS UP if (($check =~ /$validator[$a]/) && ($b == 0)) { push @html, '<hr><a href="'; push @html, @apps[$a]; push @html, '" target="_blank">'; push @html, $shortname[$a]; push @html, '</a> is <a style="color:green">up!!</a>'; if (($sslv3check == 1) || ($sslv2check == 1)) { push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>'; push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>'; push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d='; push @html, $host; push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>'; } if ($sslv3check == 1) { push @html, '<br><b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>'; } if ($sslv2check == 1) { push @html, '<br><b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; } } # WE FOUND A 404 elsif ($check =~ /404 - File or directory not found./) { $c = 1; push @html, '<hr><a href="'; push @html, @apps[$a]; push @html, '" target="_blank">'; push @html, $shortname[$a]; push @html, '</a> is showing a <b style="color:red">404 error</b>, check the server!!'; } # WE FOUND A Server Error elsif ($check =~ /An unhandled exception occurred/) { $c = 1; push @html, '<hr><a href="'; push @html, @apps[$a]; push @html, '" target="_blank">'; push @html, $shortname[$a]; push @html, '</a> is showing a <b style="color:red">Server Error</b>, check the server!!'; } # DIDNT FIND VALIDATOR, RUN OTHER TESTS else { print "The first check failed, we will wait 30 seconds then check again.\n"; # Wait 10 seconds, then run the test again, just to make sure `sleep 30s`; undef $check; $check = `curl '$apps[$a]' -m 60 2> stderr.txt`; # Skip this test if we matched something above if (($check !~ /$validator[$a]/) && ($b == 0)) { $c = 1; push @html, '<hr><p>We didn\'t find this content: <i>'; push @html, $validator[$a]; push @html, '</i> in the <b>'; push @html, $shortname[$a]; push @html, '</b> application. <b style="color:red">The application appears to be down!!</p><a href="'; push @html, @apps[$a]; push @html, '" target="_blank">Click here to check the URL</b></a><br>'; # PORT CHECK SECTION # The content we expected was not found. As long as the host exists, lets run a port check on it. if ($host ne "" ) { $ncoutput = `timeout 10s ncat -v '$host' '$port' &> ncat.tmp`; $cat = `cat ncat.tmp | grep Connected`; if ($cat =~ /Connected/) { push @html, '<h3>Summary:</h3><ul><li>From a network perspective, <b style="color:green">everything seems ok.</b></li>'; push @html, '<li>We couldn\'t find the content we expected to see. We got there, but application may not be running/installed properly.</li>'; push @html, '<li>If you see an \'Under Maintenance\' page, <b>check the inservice.txt file</b> on your server and make sure it loads locally.</li></ul>'; push @html, 'Manually check the URL (above) and verify that what is presented is expected.</p>'; if (($sslv3check == 1) || ($sslv2check == 1)) { push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>'; push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>'; push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d='; push @html, $host; push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>'; } if ($sslv3check == 1) { push @html, '<b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>'; } if ($sslv2check == 1) { push @html, '<b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; } } else { push @html, '<h3>Summary:</h3><ul><li>We couldnt find <b>content in the code</b> from the URL that we were expecting to see. </li>'; push @html, '<li>We then ran a pinch test from the Internet and it <b>also failed.</b></li></ul>'; push @html, 'Our guess is that the server is either <b>hard-down, DNS is not resolving</b> or something on the network is <b>not responding.</b><br>'; push @html, 'Next step is to view the URL from the web. If nothing loads, check the same URL local to the server.<br>'; push @html, 'If that loads you are most-likely facing a network/DNS issue.</b>'; if (($sslv3check == 1) || ($sslv2check == 1)) { push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>'; push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>'; push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d='; push @html, $host; push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>'; } if ($sslv3check == 1) { push @html, '<b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>'; } if ($sslv2check == 1) { push @html, '<b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; } } } } # APP IS UP elsif (($check =~ /$validator[$a]/) && ($b == 0)) { push @html, '<hr><a href="'; push @html, @apps[$a]; push @html, '" target="_blank">'; push @html, $shortname[$a]; push @html, '</a> is <a style="color:green">up!!</a>'; if (($sslv3check == 1) || ($sslv2check == 1)) { push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>'; push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>'; push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d='; push @html, $host; push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>'; } if ($sslv3check == 1) { push @html, '<br><b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>'; } if ($sslv2check == 1) { push @html, '<br><b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; } } } ####################################### # CERT CHECK HTML # ####################################### #If we couldn't pull a cert, prevent script from crashing if ($cert eq "") { print "We couldn't connect to the domain to run a cert check."; } else { # Convert year/mo/day to numeric date count $certnow = Date_to_Days(@certdates[0],@certdates[1],@certdates[2]); # Subtract current date and cert expiry date $certdaysleft = ($certnow - $now); #If the cert is beyond expiration, let us know if ($certdaysleft < 0) { push @html, '<br><b style="color:red">NOTE!!</b> The certificate for this domain has expired <b>' . $certdaysleft . "</b> days ago!!"; } elsif ($certdaysleft <= 14) { print "\nCert expires in " . $certdaysleft . " days!!\n"; push @html, '<br><b style="color:red">NOTE!!</b> The certificate for this domain will expire in <b>' . $certdaysleft . "</b> days!!"; } } #################################### # ---- EMAIL CONFIGURATION ---- # #################################### if ($c == 1) { print "Compiling email...please wait...\n"; $from = 'appchecker@mydomain.com'; $to = 'me@mydomain.com'; my $subject = $shortname[$a] . ' appears to be down!!'; print $subject . "Sending notification email!!\n"; my $body = '<font face="calibri"> <h2>Health Check Failed:</h2> <a href="' . @apps[$a] . '" target="_blank">Click here to check URL</a>' . ' <h4>Time of failure: </h4> ' . $time . ' A public cloud server checks for content behind this URL periodically. We ran two tests against it, and couldn\'t find what we were looking for (' . $validator[$a] . '). Click the URL to validate. After manually checking, if the page loads as you\'d expect, it may have been under heavy load at that given time. ' . 'Check historical monitoring tools for any possible outages or heavy load. </font>'; $msg = MIME::Lite->new( From => $from, To => $to, Subject => $subject, Data => qq{$body} ); $msg->attr("content-type" => "text/html"); $msg->send; } # Make sure our error parsing file is empty after each run `cat /dev/null > stderr.txt`; $a++; undef $b; undef $c; undef $check; } #################################### # ---- SFTP Configuration ---- # #################################### # # Optionally upload HTML file to an SFTP site for viewing # SFTP the file to the site of your choice # Authentication method uses keys not interactive passwords # Make sure index.html exists and is empty before we start `touch index.html`; `cat /dev/null > index.html`; # Define filename my $indexfile = 'index.html'; # Write HTML array to file open (FILE, ">> $indexfile") || die "\nProblem opening $indexfile\n"; print FILE @html; close (FILE); # Create a simple batch file to put file on your SFTP site first # Copy file to SFTP server `sftp -b mysftpbatch.bat mysftpsite.domain.com`; # Remove temporary file `rm -f ncat.tmp`; print "\nThe test is done, check the URL results\n";