Cloud-Based Web Application Check-Script

production_inspection

Monitoring applications can be tricky. As a network engineer it’s important that the applications that I serve up are available and working properly. To network engineers it’s important that resources are up and running from the Internet. This script checks a variety of aspects pertaining to a public URL.

  • Availability (TCP ports)
  • Content matching (HTTP GET)
  • Certificate Expiration
  • Security (SSL disabled, proper chain, etc.)
  • DNS (cert name mismatch, DNS lookup)

The script runs from a Linux server I have in the cloud. Use any platform you choose. All that is required is Perl, curl, openssl and netcat.The MIME::Lite and DATE::Calc Perl modules are also required.

The script will run through a list of URL’s defined by you and check for content we’d expect when the application is up. We run through a lot of other tests mentioned above. Take a look at the script and see what it checks for.

The script will push output to an HTML file that allows for “user-friendly” appearance in email alerts and the HTML file can be uploaded to a webserver.

Configure the script to run periodically with a cron job.

This is a free, easily modified script that can help monitor your applications without much overhead. Use and modify to your specific needs.

#!/usr/bin/perl
use MIME::Lite;
use Date::Calc qw(
Date_to_Days);

#############################################################
# -----VERSION NOTES-----              
# 1.0: Base configuration
# 1.1: Added color/formatting changes to output
# 1.2: Added ncat port validation after failure
# 1.3: Added email functionality
# 1.4: Modified code to send HTML instead of text
# 1.5: Save HTML output to file and SFTP to dashboard server
# 1.6: Added additional logic for failure types, e.g. 404, etc.
# 2.0: Added Email logic for detecting failures
# 2.1: Added cert checking logic


#############################################################
# To test an application, add the full URL of the app you
# want to test to the apps variable. Secondly add a string of 
# content that matches when the application is up and running. 
# If the content doesn't match, the application is marked as 
# down. 
#############################################################

# Applications short or common name goes here e.g. 'Customer WebUI'
@shortname = (	#App1
				'Google',
				#App2
				'Fake Domain'
			);
			  
# FULL application URL goes here to test for availability and content
# !!!!!MAKE SURE YOU ADD THE PORT NUMBER AFTER THE HOSTNAME FOR THE TEST TO WORK!!!!!
@apps = (	'https://www.google.com:443/',
			'https://fakedomain.domains.net:443/'
		);

# Strings to match against when you know the application is up go here. 
@validator = (	'Google Search',
				'Bad'
			);
			  
# Count number of variables in array to loop can run properly
$numapps = (scalar @apps);

# Define variables

# Needed for our while statement
$a = 0;
# If one failure check passed, don't check for other failures. Reduces redundant information.
$b = 0;
# If c = 1, app is down. Used for e-mail notifications
$c = 0;

# Define HTML array to be sent via. SFTP
my @html = ();

# Set datetime
$time = `TZ=":US/Eastern" date`;

# Add environmental header in HTML file
push @html, '<head><title>App Status</title></head>';
push @html, '

<h1>Application Status</h1>


';
push @html, '

<h3>The applications listed here are checked once every 30 minutes</h3>


';
push @html, '

Last Checked: ' ;
push @html, $time;
push @html, '

';

# Tell the user that something is happening
print "The test is running, please wait... \n";

# Begin the checks
while($a < $numapps){

 # Grab URL components to allow for more segmented testing, use variables as needed
 my($protocol, $host, $port, $uri) = $apps[$a] =~ m|(.*)://([a-zA-Z0-9\-.]+):([0-9]+)?(.*)?|;

 # Lets run some quick security tests first
 $sslv2check = `timeout 3 openssl s_client -connect '$host':443 -ssl2 2>/dev/null`;
 $sslv3check = `timeout 3 openssl s_client -connect '$host':443 -ssl3 2>/dev/null`;

 #********************
 #* CERT DATE LOGIC *
 #********************
 
 # Get todays year, month and date
 $year = `TZ=":UTC" date +%Y`;
 $month = `TZ=":UTC" date +%m`;
 $date = `TZ=":UTC" date +%d`;
 
 #Convert todays date to a numeric date count
 $now = Date_to_Days($year,$month,$date);
 
 # Grab certificate date
 $cert = `timeout 3 openssl s_client -connect $host:443 -tls1 2>/dev/null| openssl x509 -noout -enddate 2>/dev/null| cut -f2 -d'=' | xargs -0 date +%F -d 2>/dev/null`;

 # Split the output into DATE MONTH DAY
 my @certdates = split('-', $cert); 


 #End certificate expiration logic
 
 if ($sslv3check =~ /Server public key is/) {
 $sslv3check = 1;
 print "We matched the SSLv3 security check\n";
 }
 if ($sslv2check =~ /Server public key is/) {
 $sslv2check = 1;
 print "We matched the SSLv2 security check\n";
 }

 # The standard check with a 60 second timeout. This is what grabs the HTTP information response from server
 $check = `curl '$apps[$a]' -m 60 2> stderr.txt`;
 
 # If the check failed, we want to preserve that information. Stderr is saved to a variable and the checks below are ran. 
 undef $stderr;
 my $stderr = `cat stderr.txt`;
 
 # Matched DNS unresolvable 
 if ($stderr =~ /curl: \(6{1}\)/) {
 
 # I set this here so we don't run other redundant checks.
 $b = 1;
 
 print "We matched the DNS stderr log\n";
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a>: <b>We couldn\'t resolve the host that you specified</b> using public DNS servers. Something is amiss...'; 
 }
 #Matched a TCP reset
 elsif ($stderr =~ /curl: \(56\)/) {
 
 $b = 1;
 
 print "We matched the TCP Reset rule\n";
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a>: <b>We were sent a reset!!</b> Since we did not see the construction page, there are a few reasons for this:<ul>';
 push @html, '<li>The server is not accepting requests at all. Check the server locally and make sure it does the same with localhost.</li>';
 push @html, '<li>The firewall is blocking us, DNS could be wrong, or the firewall isn\'t listening on this port</li></ul>'; 
 } 
 #Matched a cert name mismatch
 elsif ($stderr =~ /curl: \(51\)/) {
 
 undef $check;
 $check = `curl -k '$apps[$a]' -m 60 2> stderr.txt`;
 
 print "We matched the Cert Name Mismatch rule\n";
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a>: <b>Requested DNS name does not match the servers certificate!!</b>';
 push @html, '<p>Check the cert that the server is providing. If the cert appears fine, make sure the utility is using the name you are expecting.';
 push @html, 'Depending on the browser, a client might not notice this, but it is best practice to fix this issue.</p>';
 }
 #Matched SSL chain failures
 elsif ($stderr =~ /curl: \(60\)/) {
 
 $b = 1;
 
 # There was an SSL issue. Lets run our curl check again without forcing chain validation and continue
 undef $check;
 $check = `curl -k '$apps[$a]' -m 60 2> stderr.txt`;
 
 print "We matched the Cert-chain stderr log, we will run curl again in insecure mode\n";
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> <b style="color:green">is up</b> with caveats... <h3>NOTE:</h3>We found an issue with the certificate chain that your server provides. Validate the chain your server is sending.<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the chain. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 push @html, 'The application may be working, but best practice we should make sure the chain your server is sending is what it should.<br>';
 
 if ($sslv3check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 } 
 }
 
 # Comparison statement to see if the content of the curl contains validator
 
 # APP IS UP
 if (($check =~ /$validator[$a]/) && ($b == 0)) {
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> is <a style="color:green">up!!</a>';
 
 if (($sslv3check == 1) || ($sslv2check == 1)) {
 push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 } 
 if ($sslv3check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 }
 }
 # WE FOUND A 404
 elsif ($check =~ /404 - File or directory not found./) {
 
 $c = 1;
 
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> is showing a <b style="color:red">404 error</b>, check the server!!';
 }
 # WE FOUND A Server Error
 elsif ($check =~ /An unhandled exception occurred/) {
 
 $c = 1;
 
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> is showing a <b style="color:red">Server Error</b>, check the server!!';
 } 
 # DIDNT FIND VALIDATOR, RUN OTHER TESTS
 else {
 print "The first check failed, we will wait 30 seconds then check again.\n";
 # Wait 10 seconds, then run the test again, just to make sure
 `sleep 30s`;
 undef $check;
 $check = `curl '$apps[$a]' -m 60 2> stderr.txt`;
 
 # Skip this test if we matched something above
 if (($check !~ /$validator[$a]/) && ($b == 0)) { 
 $c = 1;
 
 push @html, '<hr><p>We didn\'t find this content: <i>';
 push @html, $validator[$a]; 
 push @html, '</i> in the <b>';
 push @html, $shortname[$a];
 push @html, '</b> application. <b style="color:red">The application appears to be down!!</p><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">Click here to check the URL</b></a><br>';
 
 # PORT CHECK SECTION
 # The content we expected was not found. As long as the host exists, lets run a port check on it. 
 if ($host ne "" ) {
 
 $ncoutput = `timeout 10s ncat -v '$host' '$port' &> ncat.tmp`;
 $cat = `cat ncat.tmp | grep Connected`;
 
 if ($cat =~ /Connected/) {
 push @html, '<h3>Summary:</h3><ul><li>From a network perspective, <b style="color:green">everything seems ok.</b></li>';
 push @html, '<li>We couldn\'t find the content we expected to see. We got there, but application may not be running/installed properly.</li>';
 push @html, '<li>If you see an \'Under Maintenance\' page, <b>check the inservice.txt file</b> on your server and make sure it loads locally.</li></ul>';
 push @html, 'Manually check the URL (above) and verify that what is presented is expected.</p>'; 
 
 if (($sslv3check == 1) || ($sslv2check == 1)) {
 push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 } 
 if ($sslv3check == 1) {
 push @html, '<b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 }
 }
 else {
 
 push @html, '<h3>Summary:</h3><ul><li>We couldnt find <b>content in the code</b> from the URL that we were expecting to see. </li>';
 push @html, '<li>We then ran a pinch test from the Internet and it <b>also failed.</b></li></ul>';
 push @html, 'Our guess is that the server is either <b>hard-down, DNS is not resolving</b> or something on the network is <b>not responding.</b><br>';
 push @html, 'Next step is to view the URL from the web. If nothing loads, check the same URL local to the server.<br>';
 push @html, 'If that loads you are most-likely facing a network/DNS issue.</b>';

 if (($sslv3check == 1) || ($sslv2check == 1)) {
 push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 } 
 if ($sslv3check == 1) {
 push @html, '<b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 }
 }
 }
 }
 # APP IS UP
 elsif (($check =~ /$validator[$a]/) && ($b == 0)) {
 push @html, '<hr><a href="';
 push @html, @apps[$a];
 push @html, '" target="_blank">';
 push @html, $shortname[$a];
 push @html, '</a> is <a style="color:green">up!!</a>';
 
 if (($sslv3check == 1) || ($sslv2check == 1)) {
 push @html, '<br>This app accepts SSL, which is an unsecure protocol!!<br>';
 push @html, 'You can use <i>ssl-labs</i> to check the server. Here is an automated link to check yourself.<ul><li>';
 push @html, '<a href="https://www.ssllabs.com/ssltest/analyze.html?d=';
 push @html, $host;
 push @html, '&hideResults=on&latest" target="_blank">SSL-Labs Test</a></li></ul>';
 } 
 if ($sslv3check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv3 protocol. <i>Shame on you!!</i><br>';
 }
 if ($sslv2check == 1) {
 push @html, '<br><b>Note:</b> This server allows the SSLv2 protocol. <i>Extra shame on you!!</i><br>'; 
 }
 }
 }

 #######################################
 # CERT CHECK HTML #
 #######################################
 
 #If we couldn't pull a cert, prevent script from crashing
 if ($cert eq "") {
 print "We couldn't connect to the domain to run a cert check.";
 }
 else {
 # Convert year/mo/day to numeric date count
 $certnow = Date_to_Days(@certdates[0],@certdates[1],@certdates[2]);
 
 # Subtract current date and cert expiry date
 $certdaysleft = ($certnow - $now);
 
 #If the cert is beyond expiration, let us know
 if ($certdaysleft < 0) {
 push @html, '<br><b style="color:red">NOTE!!</b> The certificate for this domain has expired <b>' . $certdaysleft . "</b> days ago!!";
 }
 elsif ($certdaysleft <= 14) {
 print "\nCert expires in " . $certdaysleft . " days!!\n";
 push @html, '<br><b style="color:red">NOTE!!</b> The certificate for this domain will expire in <b>' . $certdaysleft . "</b> days!!";
 }
 }
 
 ####################################
 # ---- EMAIL CONFIGURATION ---- #
 ####################################

 if ($c == 1) {
 
 print "Compiling email...please wait...\n";
 
 $from = 'appchecker@mydomain.com';
 $to = 'me@mydomain.com';
 my $subject = $shortname[$a] . ' appears to be down!!';
 print $subject . "Sending notification email!!\n";
 my $body = '<font face="calibri">
<h2>Health Check Failed:</h2>

<a href="' . @apps[$a] . '" target="_blank">Click here to check URL</a>' . 
 '
<h4>Time of failure: </h4>

' . $time . 
 '

A public cloud server checks for content behind this URL periodically. We ran two tests against it, and couldn\'t find what we were looking for (' . $validator[$a] . 
 '). Click the URL to validate. After manually checking, if the page loads as you\'d expect, it may have been under heavy load at that given time. ' . 
 'Check historical monitoring tools for any possible outages or heavy load.

</font>';

 $msg = MIME::Lite->new(
 From => $from,
 To => $to,
 Subject => $subject,
 Data => qq{$body}
 );

 $msg->attr("content-type" => "text/html"); 
 $msg->send;
 }
 
 # Make sure our error parsing file is empty after each run
 `cat /dev/null > stderr.txt`;
 $a++;
 undef $b;
 undef $c;
 undef $check;
 
}


####################################
#  ---- SFTP Configuration ----    #
####################################
#
# Optionally upload HTML file to an SFTP site for viewing
# SFTP the file to the site of your choice
# Authentication method uses keys not interactive passwords

# Make sure index.html exists and is empty before we start
`touch index.html`;
`cat /dev/null > index.html`;

# Define filename
my $indexfile = 'index.html';

# Write HTML array to file
open (FILE, ">> $indexfile") || die "\nProblem opening $indexfile\n";
print FILE @html;
close (FILE);

# Create a simple batch file to put file on your SFTP site first
# Copy file to SFTP server
`sftp -b mysftpbatch.bat mysftpsite.domain.com`;

# Remove temporary file
`rm -f ncat.tmp`;

print "\nThe test is done, check the URL results\n";
Advertisements
Cloud-Based Web Application Check-Script

Powershell: Query domain for expiring certificates

Certificate expirations can be a pain to manage and are often overlooked. Some people have spreadsheets, set calendar reminders or just wait until a customer complains. I used to have a Unix script that would search an entire subnet for servers with expiring certs, but it was not very robust, searching subnets can return some questionable results.

This script starts by querying active directory to get a list of computer names that match string(s) that you enter. Then powershell searches the servers for certs that are within 14 days of expiring. The script will then email you an HTML report if there are certs expiring, if not it will do nothing. This script is highly customizable, so tweak it as needed.

<#

Script to check AD computers

for expiring certificates.

Author: Ryan Clark

Date: 4/1/13

#>

 

 

#Import AD module

Import-Module ActiveDirectory

 

#Make sure computers array is empty incase script has been run
in this session before

$Computers.Clear()

 

#Fill array with computers you want to filter by this example
is computers that start with DEV and PRD

$Computers = Get-ADComputer -Filter ‘Name -like “DEV*” -or Name -like “PRD*”’ | Foreach {$_.Name}

 

#Count Computers

$CompNum= $Computers.count

 

#Set a date variable for today and two weeks ago. Change the
date in TwoWeeks to modify the expiration time

$Today = (Get-Date).ToString(yyyy/MM/dd)

$TwoWeeks= (Get-Date).AddDays(+30).ToString(yyyy/MM/dd)

 

#HTML Style config

$a = “<style>”

$a = $a + BODY{background-color:white;}”

$a = $a + TABLE{border-width:1px;border-style: solid;border-color: black;border-collapse: collapse;}”

$a = $a + TH{border-width:1px;padding: 5px;border-style: solid;border-color: black;background-color:#D0A9F5}”

$a = $a + TD{border-width:1px;padding: 5px;border-style: solid;border-color: black;background-color:#FAFAFA}”

$a = $a + “</style>”

 

#Run through each computer and check for certs within the
configured date period

$i=0

Clear-content C:\Admin\CertReport.htm

while($i lt $CompNum)

{

Write-Host “Working on:” $Computers[$i]

#If your certs are in a different store change the Cert:\ path

$Certs = invoke-command ComputerName $Computers[$i] ScriptBlock {Get-ChildItem Cert:\LocalMachine\My}

$CertCount=$certs.count

$j=0

 while($j lt $CertCount)

 { 

   if (($Certs[$j].NotAfter gt $Today) -and ($Certs[$j].NotAfter lt $TwoWeeks) )

   {

    $Certs[$j] | ConvertTo-Html -head $a -title “Expiry Information” -property PSComputerName,Subject,NotAfter
>> C:\Admin\CertReport.htm

   }

   ++$j

 }

++$i

}

 

#Either email a report or do nothing

if
((Get-Content “C:\Admin\CertReport.htm”eq $Null)

{

 write-host
“No expiring certificates. Ending script.”

}

else

{

 #Modify the -to field to send to another user or DL

 write-host
“Expired certificates found, emailing report”

 Send-MailMessage -to me@mydomain.com -Subject
“Certificate Report” SmtpServer x.x.x.x -From myserver@domain.com -Attachments C:\Admin\CertReport.htm

}

Powershell: Query domain for expiring certificates

Autodelete Files by Age

In my last post I covered how to back up files based on age. This is a nice script to supplement it, this will auto-delete files based on their age. It’s an easy way to backup files to a disk somewhere and not overflow it with backups.

This example is for a Windows host, to delete a file older than 3 days:

1. Create a batch script with the following

echo on
 rem Delete files older than 3 days
 FORFILES /P C:\Admin\Test\ /S /M 1*.bmp /D -3 /c " CMD /c del /q @FILE "

2. Modify the following flags to suit your need

/p = The path to search for the files you want to check the date of and remove
/s = Recurse subdirectories contained within the path specified using /p and check them as well
/m = The search mask to be used for the file type you want to check the date on (*.* being all files)
/d = The date to compare the files against. A standard date type can also be used (dd/mm/yyyy)
/c = The command to be used on a file that matches the /m and /d criteria
/q = Used within /c to instruct the del command to delete files quietly

3. Add the batch file to the scheduler based on your need

Autodelete Files by Age