Create Software Load-Balancer using nginx

nginx-logo

I wrote an article a few years ago with instructions on how to build a software load-balancer with nginx, haproxy and stunnel. Needless to say, it was a pain in the butt. Those technologies were not super mature and it took a lot of work to get things going. Luckily nginx is a lot further along than it was back in 2011 when I created that setup. Now everything that I needed back-then can be done with nginx!!

The important part to take into consideration is this configuration supports end-to-end SSL which combines SSL termination (front end) with SSL initiation (back end), virtual directory (uri) matching which points URI matches to specific serverfarms or pools and http to https redirection. That’s a mouthful, but to me that type of configuration is pretty standard these days.

These steps are based off a configuration that I did using a CentOS 6.6 image running in Azure cloud. The steps assume you have a similar RedHat based distribution with a static public IP (NAT’ed hopefully) and an internal IP that can communicate to the internal server network.

 Update the server

# yum update
# reboot

 Add the nginx Repository

# vi /etc/yum.repos.d/nginx.repo

Add the Following

[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

 Validate Repository

The nginx repository will be listed.

# yum repolist

 Install nginx

# yum install nginx

Request Certificate

Enter the information as OpenSSL prompts you, then submit the csr to the CA that you prefer.

# openssl req -nodes -newkey rsa:2048 -keyout /etc/nginx/myserver.key -out /etc/nginx/server.csr

Once you get your cert bundle back from the CA, [lace the certificate file and intermediate certificate in the directory of your choosing (I’m using /etc/nginx to simplify things).

Merge the intermediate cert with your server cert.

# cat myserver.crt intermediate.crt >> mycertificate.crt

Modify your nginx.conf file

Here is where things are going to depend on your specific application. The example I will paste below will show a basic example that covers the functions above. Add or modify to suit your needs.

user nginx;

#This server has two processors
worker_processes 2;

#Assigning each worker to a processor
worker_cpu_affinity 01 10;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
 worker_connections 1024;
}

http {
 include /etc/nginx/mime.types;
 default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
 '$status $body_bytes_sent "$http_referer" '
 '"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on; 
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;

#Define default server pool
upstream DEFAULT {
 server 192.168.10.5:443;
}

#Define a secondary application server
upstream APP1 {
 server 192.168.10.6:443;
}

#Define a third application server
upstream APP2 {
 server 192.168.10.7:443;
}

#Create an http to https redirect
server {
 listen 192.168.10.10:80;
 return 301 https://$host$request_uri;
}

server {

 listen 192.168.10.10:443;
 server_name mydomain.com;

 ssl_certificate /etc/nginx/mycertificate.crt;
 ssl_certificate_key /etc/nginx/myserver.key;

 ssl on;

 ssl_session_cache builtin:1000 shared:SSL:10m;
 ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
 ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
 ssl_prefer_server_ciphers on;

#Define application 1
 location ~* ^/application1 {
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;
  proxy_redirect off;
  proxy_max_temp_file_size 0;

 proxy_pass https://APP1;
 }

#Define application 2
 location ~* ^/application2 {
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;
  proxy_redirect off;
  proxy_max_temp_file_size 0;

 proxy_pass https://APP2;
 }

#Default catch-all
location ~* ^/(.*) {
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header Host $http_host;
 proxy_redirect off;
 proxy_max_temp_file_size 0;

 proxy_pass https://DEFAULT;
 }
}
}

Restart Service & Test Application

If your server doesnt start properly, check log files in /var/log/nginx/error.log

# service nginx restart

If nginx starts properly, check your public DNS name via HTTP to validate the redirect works, then the virtual directories to make sure pattern-matching is working.

Advertisements
Create Software Load-Balancer using nginx

Deploy LAMP & phpMyAdmin on RHEL 6.5

lamp

I know there are a ton of articles out there on this topic. But I had a hard time finding one for RHEL 6.5 (NOT CentOS). There are subtle differences that lead to undoubted failure if you solely follow instructions meant for CentOS. So for all you RHEL people out there tasked with this, here you go.

It may or may-not work on different versions/distributions–no guarantee. I’m assuming you’ve installed the base image from ISO with no additional options and are waiting at the command line with privilege and Internet access.

Get your Repositories

Register your box with RHN.

# subscription-manager register --username user --password password

EPEL is required to properly install phpMyAdmin.

# wget http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
# rpm -ivh epel-release-6-8.noarch.rpm

The ‘optional’ repository is also needed to properly resolve dependencies.

# yum-config-manager --enable rhel-6-server-optional-rpms

Validate your repositories. Something like this should be displayed:

# yum repolist
repolist

Prep Steps

Update Server
# yum update -y
# reboot
Disable Firewall

I like to disable the local firewall, I use my hardware firewall for that. Modify iptables if the local firewall is something you want to do. There’s plenty on that out there.

# service iptables stop
# chkconfig iptables off
Install Apache
# yum install -y httpd
# service httpd start
# chkconfig httpd on
Validate Apache
# curl localhost | grep test

The HTML content will be visible from the test page here if everything went well.

Install PHP

Optionally install php-mysql if your application requires MySQL support with PHP.

# yum install -y php
# yum install -y php-mysql (optional)

Create a test PHP page.

# echo '<?php phpinfo() ; ?>' > /var/www/html/test.php

Validate the content loads. The command will display content based on your PHP config and versions.

# curl localhost/test.php

Remove the test page for security purposes

# rm -f /var/www/html/test.php
Install and start MySQL
# yum install -y mysql mysql-server
# service mysqld start
# chkconfig mysqld on

Secure and setup MySQL by running the vendor script. Press Enter at each prompt and set the root password when asked.

# mysql_secure_installation
Install phpMyAdmin (Optional)

PHPmyadmin is a GUI frontend for MySQL database management.

# yum install -y phpmyadmin

Configure remote access to phpmyadmin by editing the following file:

# vi /etc/httpd/conf.d/phpMyAdmin.conf

Do a search and replace for 127.0.0.1 and change to the IP you will use to manage MySQL through a browser, or add a subnet. In my case everything on my internal network starts with 10.x.x.x, so I did the following.

<Directory /usr/share/phpMyAdmin/>
   AddDefaultCharset UTF-8
   <IfModule mod_authz_core.c>
     # Apache 2.4
     <RequireAny>
       Require ip 10.0.0.0/8
       Require ip ::1
     </RequireAny>
   </IfModule>
   <IfModule !mod_authz_core.c>
     # Apache 2.2
     Order Deny,Allow
     Deny from All
     Allow from 10.0.0.0/8
     Allow from ::1
   </IfModule>
</Directory>
<Directory /usr/share/phpMyAdmin/setup/>
   <IfModule mod_authz_core.c>
     # Apache 2.4
     <RequireAny>
       Require ip 10.0.0.0/8
       Require ip ::1
     </RequireAny>
   </IfModule>
   <IfModule !mod_authz_core.c>
     # Apache 2.2
     Order Deny,Allow
     Deny from All
     Allow from 10.0.0.0/8
     Allow from ::1
   </IfModule>
</Directory>

Change authentication type to http.

# cp /usr/share/phpMyAdmin/config.sample.inc.php /usr/share/phpMyAdmin/config.inc.php 
# vi /usr/share/phpMyAdmin/config.inc.php

Change cookie to http

[...] 
/* Authentication type */
$cfg['Servers'][$i]['auth_type'] = 'http';
[...]

Finish Up

I like to reboot for good measure, making sure your services start on their own and are functional.

# reboot

Finally, open a browser on the machine you allowed in the steps above and browse to the following:

http://mylampserver/phpmyadmin 

Log in with the root user you created for MySQL.

That’s it, enjoy!!

Deploy LAMP & phpMyAdmin on RHEL 6.5

F5 Persistence Mirroring w/ iRules

img_1926

In a failover pair of devices, mirroring persistence records is important. When one LTM fails, we need the other to know which server a user’s session was “stuck” to. If a device failed and the other device was not aware of these records, the decision process would start again and a user could lose their session (wait for the tickets to roll in…) We want our HA pair of F5’s to failover seamlessly so they can be updated without impacting users, as well as transparency to end-users when a device fails.

The problem with the F5 is when you are using multiple persistence methodologies, mirroring does not function for each type you are using in the iRule, only the type in the ‘Default Persistence Profile’ chosen in the virtual-server. This is a problem when you are using a few different types in one iRule. For example; persist cookie, persist source_addr, persist uie, etc. all in one iRule.

So what do I do? The solution to this is difficult if not impossible to find in dev-central. I had to open a support case to even find this out. Use universal persistence!! In almost all cases when I opened support tickets the engineer would state that when I was using ‘persist cookie’ that it was unusual. I always thought to myself, “What, this is crazy!! Why even offer me a persistence type if it is not recommended to be used by the manufacturers of the device themselves!!”

The solution to this problem is to convert your existing iRule that uses a few different types of persistence to all use universal. The syntax may not be as simple, but universal persistence can do everything the other types can and more. Unfortunately F5 doesn’t have universal persistence documented very well (common theme), but this is a good start.

Let’s go over a simple example iRule. In this example I’m examining URI (virtual directory) and making decisions based on that value. I am using source IP and cookie hash stickiness. Different apps require different types of persistence. A normal person would offer what F5 provides, but that is not recommended by F5.

when HTTP_REQUEST {
 switch -glob [string tolower [HTTP::uri]] {
 "/app1*" {
 pool POOL-APP1
 persist source_addr 255.255.255.255
 }
 "/app2*" -
 "/app3*" {
 pool POOL-APP23
 persist cookie hash "ASP.NET_SessionId"
 }
 "/app4*" {
 pool POOL-APP4
 persist none
 }
 default {
 HTTP::redirect https://[getfield [HTTP::host] ":" 1][HTTP::uri]
 }
 }
 }

Unfortunately I can only pick one default persistence type in my virtual-server so not every record will fail over to the second node. We need to fix this by converting the syntax to universal and pick UIE as the default type.

when HTTP_REQUEST {
 switch -glob [string tolower [HTTP::uri]] {
 "/app1*" {
 pool POOL-APP1
 persist uie [IP::client_addr]
 }
 "/app2*" -
 "/app3*" {
 pool POOL-APP23
 persist uie [HTTP::cookie "ASP.NET_SessionId"]
 }
 "/app4*" {
 pool POOL-APP4
 persist none
 }
 default {
 HTTP::redirect https://[getfield [HTTP::host] ":" 1][HTTP::uri]
 }
 }
 }

Pretty simple, no? Now I assign my universal persistence profile with the ‘Mirror Persistence’ option checked.

uie1

Persistence records will now mirror to the passive device. View the records on each device by running ‘tmsh show ltm persistence persist-records’ to validate that the records are on each F5.

Aside: I think by now you all know how big of a fan I am of the Cisco ACE. Guess how easy this is with the ACE? Add ‘replicate sticky’ to your sticky group configuration. Way too simple…

F5 Persistence Mirroring w/ iRules

Poodle Vulnerability: Who’s Using SSLv3?

poodle

There are a ton of different articles on devcentral about certain iRules for changing the encryption that users are using, dropping them, etc. I found it difficult to find any code specific to just logging who is using SSLv3 so I can track them down and try to contact the customer. Managers found the reports that I generated useful to make a business decision instead of simply shutting of SSLv3 on the F5.

All this rule does is define a high-speed-logging server and send a message to the logging server so you can run reports against the logs. It logs source IP and URI (virtual-directory) from the request. Apply this to whichever virtual-server you need to log these requests from.

when CLIENT_ACCEPTED {
   set logpub [HSL::open -publisher /Common/HSL-SERVER]
}
when HTTP_REQUEST {
   set cipher [SSL::cipher version]
   set uri [HTTP::uri]
   if { $cipher equals "SSLv3" } {
      HSL::send $logpub "$cipher encryption used from client; [IP::client_addr] to $uri"
   }
}
Poodle Vulnerability: Who’s Using SSLv3?

iRule – IP Restrict Internal Applications

Fork in the road

In some scenarios it is necessary to limit access to applications behind the F5. Lets say you are hosting an application that is load balanced and is publicly and privately accessible on the same virtual-server. This can be accomplished by using an iRule.

In my scenario I need to limit access to applications running on an HTTP (non-SSL/TLS) virtual-server so that internal clear-text calls are permitted on the LAN, but users on the Internet are redirected to the same resource on HTTPS. My private networks all start with 10.* so it was fairly easy for me to write a rule to accomplish this. The rule can be customized to fit many different scenarios.

In summary my logic first inspects the IP address of the clients request. If it starts with 10., process the request by looking at URI (virtual-directory). If there is a successful match, send to the appropriate server. If not, redirect the full HOST and URI request from the client to HTTPS.

Secondarily if the clients IP does not match my public range, drop requests to certain resources that are not accessible at all from the Internet and redirect the rest to HTTPS as we did before.

 when HTTP_REQUEST {
    if { [IP::client_addr] starts_with 10. } {
       switch -glob [string tolower [HTTP::uri]] {
          "/app1*" -
          "/app2*" {
                pool POOL-APP-HTTP
                persist none
          }
          "/fs*" {
                pool POOL-FS-HTTP
                persist none
          }
          default {
                HTTP::redirect https://[getfield [HTTP::host] ":" 1][HTTP::uri]
          }
      }
 }
 else {
    switch -glob [string tolower [HTTP::uri]] {
       "/internal1*" {
          drop
       }
       default {
          HTTP::redirect https://[getfield [HTTP::host] ":" 1][HTTP::uri]
       }
    }
 }
}
iRule – IP Restrict Internal Applications

Language Aware Maintenance Page

It seems to be an expectation now that web-applications are language and location-aware. When a web application fails, the F5 can present a ‘sorry’ page using a variety of methods. I’ve found that the iFile presentation of HTML files using iRules to be the easiest for me.

In addition to detecting the health of members in a pool, this iRule includes the detection of the users browser Accept-language to forward them to the correct language sorry page. Assign this iRule to your virtual server and it will start working once a pool has zero healthy nodes in it.

The following iRule checks to see if there are less than one member in a pool. If this is the case–the header in request is then checked to check the Accept-Language and present the correct HTML page based on that field. I check for French, Japanese and the default is English. Add additional languages as needed and their respective HTML page to present to users. Of course you will need to create and import your maintenance page into the F5 and then add it to the iRule iFile list.

when HTTP_REQUEST {
 if {[active_members [LB::server pool]] < 1} {
 log local0.crit "Server Pool: [LB::server pool] has failed"
 switch -glob [HTTP::header "Accept-Language"] {
 "fr*" {
 HTTP::respond 200 content [ifile get maintenance.fr.html] "Content-Type" "text/html"
 }
 "ja*" {
 HTTP::respond 200 content [ifile get maintenance.jp.html] "Content-Type" "text/html"
 }
 default {
 HTTP::respond 200 content [ifile get maintenance.en.html] "Content-Type" "text/html"
 }
 }
 }
}
Language Aware Maintenance Page

Rewriting IIS Server Response Content from HTTP to HTTPS

eraser-paper

This may seem like an unusual post, but this may apply to “load-balancer gurus” who do SSL termination. In certain cases web-applications that you host have embedded code that reference http links. This can be resolved with F5 iRules, but it can be tricky with Cisco ACE.

Instead of reinventing the wheel, here’s how to do it with the URL Rewrite module in IIS. A good reference link can be found here.

Lets say your webserver keeps responding with image referrer links with http instead of https. This will give users on the web errors about mixed content. IIS can rewrite those responses for you without spending tons of man-hours fixing your code or bothering developers.

What this is doing is looking for any content that starts with HTTP and grabs any content after the star as a back-reference to basically plop into use in the next portion. {R:2} is that reference in the rewrite value.  I’m looking for http patterns after img, link and script tags. You can customize to your specific application.

 <rule name="Content-Rewrite" preCondition="" enabled="true" patternSyntax="Wildcard">
  <match filterByTags="Img, Link, Script" pattern="*http://*" />
   <conditions />
    <action type="Rewrite" value="https://{R:2}" />
  </rule>
  <preConditions>
  <preCondition name="http" logicalGrouping="MatchAny" patternSyntax="Wildcard">
  <add input="{RESPONSE_CONTENT_TYPE}" pattern="http:" />

Rewriting IIS Server Response Content from HTTP to HTTPS