Load balance based on CPU Load – (Windows 2008 Hosts)

Some load-balanced applications create considerable load independent to number of connections. For example reporting a server may become overloaded if users submit a report that requires a lot of historical data to generate. If there are three servers in a farm and they each have 10 connections, one server could have people running intensive reporting, while the other is sitting idle, which is imperceptible to the ACE with out some sort of CPU inspection. This is where SNMP probes come in…

The first thing we need to do is enable SNMP services. In my case I’m using Windows Server 2008 hosts. On Unix or other OS’s your process will vary or it may already be running SNMP services.

  • Open Server Manager
  • Click Features
  • Click Add Features on the right side
  • Scroll down and check the box marked SNMP Services
  • Click Next then Install

The next thing you need to do is configure it so that the ACE can talk to the host with the right strings.

  • Go back to Server Manager and expand Configuration
  • Click Services
  • Right click on SNMP Service and click Properties
  • Click the Security tab
  • Uncheck Send authentication trap
  • Under Accepted community names click Add...
  • Leave READ ONLY
  • Enter a community name appropriate for your environment (or use public) and click Add
  • Under Accept SNMP packets from these hosts click Add
  • Enter the IP of the ACE and click Add
  • Right click the service and click Restart

Now the ACE can poll the server for SNMP entries.

The first step is configuring the rserver(s) that are going to be monitored.

rserver host SERVER1
  ip address 192.168.1.1
  inservice
rserver host SERVER2
  ip address 192.168.1.2
  inservice

Now we want to build out our SNMP probe. The first step is to define it.

probe snmp CPU-PROBE

What was your community name? Enter it next.

community public

How often do you want the ACE to check the CPU? I used 10 seconds.

interval 10

If the server goes down, how many successful probes before it comes back online? Six would be 60 seconds, so I’ll use that.

passdetect count 6

The next section is tricky. How many CPU’s does your server have? You’ll have to customize your probe based on the number of CPU’s. In my case my server has two CPU’s. For one CPU the oid is .1.3.6.1.2.1.25.3.3.1.2.2 and the other is .1.3.6.1.2.1.25.3.3.1.2.3

What I’ll need to do is add both OID’s to the probe, then give them equal weight. In my example each CPU has a weight of 8000 (all your OID’s have to add up to 16000). If you had 8 CPU’s the weight would be 4000 each, and so on.

oid .1.3.6.1.2.1.25.3.3.1.2.2
 weight 8000
oid .1.3.6.1.2.1.25.3.3.1.2.3
 weight 8000

Now you can assign it to a serverfarm as a predictor method.

serverfarm MYFARM
  predictor least-loaded probe CPU-PROBE
  rserver SERVER1
    inservice
  rserver SERVER2
    inservice

The load is computed with the total weight of the probe, which is 16000. Run the show probe CPU-PROBE detail command to view the load on the server. Take that number and divide by 16000 to get the percentage value.

Here are some examples, the first one is for two-CPU servers and the second is for six-CPU servers.

Two CPU

probe snmp CPU-PROBE-TWO-CPU
 interval 10
 passdetect interval 60
 passdetect count 6
 community public
 oid .1.3.6.1.2.1.25.3.3.1.2.2
 weight 8000
 oid .1.3.6.1.2.1.25.3.3.1.2.3
 weight 8000

Six CPU

probe snmp CPU-PROBE-SIX-CPU
  interval 10
  passdetect interval 60
  passdetect count 6
  community public
  oid .1.3.6.1.2.1.25.3.3.1.2.2
    weight 2667
  oid .1.3.6.1.2.1.25.3.3.1.2.3
    weight 2666
  oid .1.3.6.1.2.1.25.3.3.1.2.4
    weight 2667
  oid .1.3.6.1.2.1.25.3.3.1.2.5
    weight 2667
  oid .1.3.6.1.2.1.25.3.3.1.2.6
    weight 2666
  oid .1.3.6.1.2.1.25.3.3.1.2.7
    weight 2667
Load balance based on CPU Load – (Windows 2008 Hosts)

HTTP to HTTPS Redirect on Cisco ACE

Instead of always having to write an HTTPS redirect on your webservers, you can have the ACE handle this work for you.

The first step is to create your redirection “rservers”. In my example it matches everything. If you want to only redirect a specific URL, then change the content after “https” to match what you want.

rserver redirect HTTPS-REDIRECT
  webhost-redirection https://%h%p 302
  inservice

Add to a redirect serverfarm:

serverfarm redirect FARM-HTTPS-REDIRECT
  rserver HTTPS-REDIRECT
     inservice

Once you have the serverfarm configured, apply to the respective policy-map. To match everything apply it to a class-default, if you want to redirect a specific URL assign it against a policy-map with multiple class definitions.

policy-map type loadbalance first-match POLICY-HTTPSREDIR
  class class-default
     serverfarm FARM-HTTPS-REDIRECT

Apply the policy map to your multi-match policy.

policy-map multi-match POLICY-STG1
 class CLASS-MYIP
   loadbalance vip inservice
   loadbalance policy POLICY-HTTPSREDIR
   loadbalance vip icmp-reply
 class SNAT
   nat dynamic 1 vlan 100

Add that to your interface and you’ll be good to go.

HTTP to HTTPS Redirect on Cisco ACE

ACE–why certificate names matter!!

I learned a lesson last night helping a co-worker with an ACE problem. The lesson was the importance of certificate names on ACE load balancers–specifically in an active/passive configuration.

Lets say you define your proxy service like this:

ssl-proxy service MyProxy
  key MyKey-2048
  cert My_Fun_cert.crt
  chaingroup MyChain
  ssl advanced-options MyCipher

In this configuration the certificate is called My_Fun_cert.crt. This exists on the ACE where it was defined and everything is happy. However, it may not on the passive ACE (our passive ACE the cert was a different name). This is easier to overlook when upgrading certs. If you upgrade and forget to upgrade the passive as well, big trouble is on the way…

In the scenario that I witnessed; the active ACE had its certificate upgraded, the passive did also, but it was named differently.

The primary ACE that was running happily for months decided to crash, causing a failover to the passive node. When this happened the certificate defined in the shared configuration did not exist in the passive ACE. Uh oh!! If you opened a browser connection to the VIP, specifically Firefox, you got the following error:

ssl_error_rx_record_too_long

Kind of cryptic, but it told me there was a problem with SSL proxying.

We realized that the certificate was in-fact updated, but it did not match what was on the primary ACE (sh crypto files, show crypto certificates file). Once that was straightened out:

crypto export WrongNameCert.crt
--Copy to notepad and paste to:
crypto import RightNameCert.crt
crypto delete WrongNameCert.crt

When the passive became the primary and things were not working, we moved again to the primary. Still no luck… After some time of analyzing the config, we realized that the proxy service definition was removed from all policy-maps!

So when the passive became the active without the right certificate names, the running-configuration was changed by the ACE.

The end-solution was to

  1. Fix the problem with the certificate names
  2. Add the correct certificate name to the proxy-service
  3. Re-add the proxy-service definitions to the policy-maps that need SSL encryption.
ACE–why certificate names matter!!

Configure SSL Termination on ACE

There are three methods of SSL proxying on the ACE; SSL Initiation, SSL termination and end-to-end SSL. In this article I am covering SSL termination.

SSL Termination

This method makes the ACE do the heavy lifting. Extra CPU cycles to encrypt/decrypt SSL traffic are offloaded to the ACE. In the tests that I have done in our development environments–it has given us, on average, a 10% overall performance increase vs. doing end-to-end SSL encryption with IIS 7.

In this case, the backend servers (IIS, Apache, etc.) do not have certificates on them. They are running port 80 (or any other cleartext port) behind the ACE. The default gateway will be either the ACE (one-arm mode) or the firewall (bridged mode) for your hosts.

NOTE: Some applications will send redirects out to the web-clients. If these redirects are not intercepted by the ACE and changed to HTTPS, it will redirect your clients to a port that is most likely not open on the firewall. I wrote an article that covers that issue here.

The only caveats with this methodology is that your clients do not have requirements against traffic being clear-text on the local network. No un-encrypted traffic traverses the Internet, but some customers and certifications do not allow any clear-text transmission of data.

What Serverfarms?

Do all your serverfarms need SSL or do just a few need SSL? Take note of what needs SSL at this point. For my example my SSL serverfarm is as follows:

serverfarm host FARM-SSL
 rserver SRVR1
   inservice
 rserver SRVR2
   inservice
 rserver SRVR3
   inservice
 rserver SRVR4
   inservice

Generate & Import Certificates

I covered how to do this on a previous post. You can find it here.

Create a VIP for SSL Traffic

What IP will you NAT on the firewall for SSL traffic? Pick one now.

class-map match-any CLASS-IP-SSL
  2 match virtual-address 10.100.1.100 tcp eq https

Create SSL Proxy

You created your key, imported your certs and configured your chaingroup already right? If not follow the steps above.

ssl-proxy service SSL-PROXY
  key MYKEY
  cert My-CERT-2012
  chaingroup MYCHAIN
  ssl advanced-options CIPHER

Easy right? I find this easier than Windows once you get the hang of it.

Create Policy to match Serverfarm

Now you want to create a policy that defines what serverfarm(s) are going to be behind the SSL proxy. In our case we only want one. If you had multiple you would simply add the class-maps under this policy.

policy-map type loadbalance first-match POLICY-SSL
  class class-default
    serverfarm FARM-SSL

Put it all together

Write your policy-map that puts everything together that you can assign to an interface.

policy-map multi-match VIP-SSL
  class CLASS-IP-SSL
    loadbalance vip inservice
    loadbalance policy POLICY-SSL
    loadbalance vip icmp-reply

Assign it to an Interface

In my case I have VLAN interfaces for bridged mode. So I assign it to both, one so the firewall can NAT it and one so the clients can hit the VIP.

interface vlan 100
  description Server Side VLAN
  bridge-group 1
  access-group input PERMIT-ALL
  service-policy input VIP-SSL
  no shut 
interface vlan 200
  description FW Side VLAN
  bridge-group 1
  access-group input PERMIT-ALL
  service-policy input VIP-SSL
  no shut 

interface bvi 1
 ip address 10.100.1.253 255.255.255.0
 description ACE IP Address
 no shutdown

That’s it! Now you should be able to hit your VIP IP from your server subnet and it should load the site with SSL. If it doesn’t, check your serverfarms, gateways, listening ports on the hosts, etc.

Configure SSL Termination on ACE

Easiest way to request an ACE Certificate

There are a variety of different ways to generate a CSR, request a certificate and import that cert into the ACE. I’ve found the easiest way to do this is through the console. No need to set up an FTP/TFTP server, just copy and paste. This is a pre-requisite to SSL-proxying on the ACE.

  • Create your CSR parameters.

CSR parameters are what will be pre-populated in your certificate request. The most important of these is the “common-name” which will be the domain your certificate is assigned to.

  1. Define csr-params 
crypto csr-params MY-PARAM
  country US
  state NY
  locality City
  organization-name Acme
  common-name example.domain.com
  serial-number 1000

There are other csr-parameters you can define if needed:

switch/Context1(config-csr-params)# ?
Configure CSR parameters:
common-name  Configure organization's common name (required parameter)
country      Configure country code (required parameter)
email        Configure email address
locality     Configure locality name
organization-name  Configure organization name
organization-unit  Configure organization unit's name
serial-number      Configure serial number
state              Configure state name (required parameter)

2. Define cipher set

Your cipher set defines the level of encryption for your site. Some clients require a specific level of encryption, while others do not. Perhaps you need to comply with some business certification (SAS70, SOX, etc.) that controls the overall level of your ciphers. Check with your business or clients for what to define this as.

The session-cache timeout is pretty self explanatory. Some users want it longer than the default. It’s good to define this so when users ask you can always check the config.

parameter-map type ssl CIPHER
  cipher RSA_WITH_RC4_128_SHA
  cipher RSA_WITH_AES_128_CBC_SHA
  cipher RSA_WITH_AES_256_CBC_SHA
  session-cache timeout 600

3. Generate your RSA key pair.

switch/Context1# crypto generate key 2048 MYKEY

I always choose a 2048 bit key because many certificate authorities require at least a 2048 bit key. For many CA’s, 1024 used to be the minimum but many have moved up to 2048.

4. Generate your CSR.

switch/Context1# crypto generate csr MY-PARAM MYKEY
-----BEGIN CERTIFICATE REQUEST-----
MIICeTCCAWECAQAwNDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAk5ZMRgwFgYDVQQD
Ew90ZXN0LmRvbWFpbi5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQC0ArGUgdcqwgCEZNREvPGOL+rEzZBmoQN/rE+Nawr0KpMIFzcPEV2Bli9hTUuk
j9L0Q/I2VyuFFHUvlIcf+ZO8gS764sPkodKAWjrWKgvK8zAwlJih02cl//mn8zWO
HRJ30LxAXRXsbbMH1sBLwZIusIJcBDqbmAyhbmvAxFco5TzQ3D+RbBXVNPw7u/cv
NM6DH2S60CgCorbU+5AxWY0vPAUOpBrae9nRPqnEe6w2/khzRW6w/HjEPciQrHbj
qV07e071UUsxh`XJegL92SaZV2J4Kl746CheA1rAchncx2NHS4bJUfgsnyztMLus
A6G2/kxK/R0EAUwUeBB6tzYBAgMBAAGcADANBgkqhkiG9w0BAQQFAAOCAQEAR12z
KBmONU/qujbhaz/2jWMDTAq2JXvX+WjgIxKNLAn45tkkOFnmnUVQLCegDfH5/roO
AIEX539TnpJfKaFQPmV7Xdq/GX/Xn/5OJkpXhQ7nQF0nhGSh8ZAazn2sCOpvQ9aQ
jsnBJVr9VrLEgGWuthIc+lUCHaXSLWj8QOSVxhcTNtlpFIfIShniy9AbFDDwRefU
J4hH9A+SGRBSF+JDzN25E+KGkEpBkPl+yot8xzWxQSO9ADN2ZBEauE7vrPy8KeJv
Rsm/3TjJQncFkgXS6NH5CL0S25dyrKSGhwG0GTVWORKiSZAyF/jxdoD+Q0C/AHvg
gsqLVoC8xg4OmM+AqA==
-----END CERTIFICATE REQUEST-----

You will be presented with the CSR. If you are using putty this is easy. Just select everything from BEGIN to END and paste into a plain-text file. Name it MYCSR.csr or whatever relates to what you are requesting.

5. Submit your CSR to a CA (godaddy, comodo, etc.)

6. Once your certificate request has been approved you should get an email or ZIP file containing three things:

  • Root certificate
  • Intermediate certificate
  • Your certificate

7. Import root, intermediate and your certificate. Again the easiest way to do this is in putty.

  • Open your root certificate in notepad. Copy it.
switch/Context1# crypto import terminal ROOT-CERT

Paste the cert.

  • Open your intermediate certificate in notepad. Copy it.
switch/Context1# crypto import terminal INTERMEDIATE-CERT

Paste the cert.

  • Open your server certificate in notepad. Copy it.
switch/Context1# crypto import terminal My-CERT-20xx

Paste the cert.

8. Now that all your certificates are imported, you’ll want to create a chaingroup with root and intermediate certs.

crypto chaingroup MYCHAIN
  cert INTERMEDIATE-CERT
  cert ROOT-CERT

9. Verify that your certificates are stored on the ACE correctly.

switch/Context1# sh crypto files
Filename              File File Expor Key/
                      Size Type table Cert
-----------------------------------------------------------------------
INTERMEDIATE-CERT     1509 PEM  Yes   CERT
MYKEY                 1679 PEM  Yes   KEY
ROOT-CA-CERT          1740 PEM  Yes   CERT
My-CERT-2012          2045 PEM  Yes   CERT
cisco-sample-cert     1082 PEM  Yes   CERT
cisco-sample-key      887  PEM  Yes   KEY

Now your certificates are on the ACE! All you need to do now is set up a proxy service.

Easiest way to request an ACE Certificate

ACE URL Redirects

Sometimes a web applications virtual directory can change. You may have many rservers behind your load-balancer that would require a configuration change to send the redirect. In this case, it’s easier to configure the redirect right on the ACE instead of the rservers web-server (IIS, Apache, etc.) instead.

So lets say your website is http://site.domain.com/oldsite and the application team built a new app and wants everyone to be redirected to http://site.domain.com/newsite

First you need to define the redirect:

rserver redirect REDIRECT1
  webhost-redirection http://%h/newsite 301
    inservice

The %h above inserts the host header from the request, which in our case would have been site.domain.com. You can also use %p which inserts the full URL path string from the request. This most likely wont be the one you use.

  • %h—Inserts the hostname from the request Host header
  • %p—Inserts the URL path string from the request

The 301 means the resource has been moved permanently. If it’s just a temporary change, you can use 302. Sometimes admins just use 302 because it’s the default.

  • 301—The requested resource has been moved permanently. For future references to this resource, the client should use one of the returned URLs.
  • 302—(Default) The requested resource has been found but has been moved temporarily to another location. For future references to this resource, the client should continue to use the request URI because the resource may be moved to other locations occasionally.

Now your redirection rule is configured. You need to assign it to a serverfarm. Instead of a host serverfarm, you want to define a redirect serverfarm.

serverfarm redirect FARM-REDIRECT
  rserver REDIRECT1
    inservice

So you’ve got a serverfarm and a rserver redirection configuration. Now it’s time to define the classmap that will match the virtual directory the user requests.

In our example the user is going to /oldsite; in many cases you already have this configured–since we are doing a redirection to begin with.

class-map type http loadbalance match-any CLASS-MAP-APPS
  5 match http url /oldsite.*
  6 match http url /hello.*
  7 match http url /osiris.*

So we want to write a new classmap that will match only the virtual directory we want users to be redirected from. This way we don’t mess up any of the current virtual-directory matches we have defined.

class-map type http loadbalance match-any CLASS-MAP-REDIRECT
  5 match http url /oldsite.*

Now we’re pretty much ready to go. Add the new class-map to your policy-map:

policy-map type loadbalance first-match MY-POLICY
  class CLASS-MAP-APPS
    serverfarm FARM-APPS
  class CLASS-MAP-REDIRECT
    serverfarm FARM-REDIRECT

Now everything is configured for the new redirection. BUT the old class map virtual directory match is still there. We need to remove it to make our changes go into place. This makes our changes immediate.

class-map type http loadbalance match-any CLASS-MAP-APPS
  no 5 match http url /oldsite.*

Now test your site! I like to use Chrome with it’s developer tools to capture each request. You should see the initial request, then the newsite coming back with a 301.

ACE URL Redirects

URL Rewrite for SSL Termination

Lets say you are planning on doing SSL termination on your ACE load balancer. This involves the SSL transactions being handled by the front-end of the ACE instead of your server. This speeds things up considerably for the backend servers, offloading SSL processing to the ACE.

In most cases your applications will have at least a few redirects written in them. If your backend server is using cleartext (http) only, the redirects will begin with http:// thus directing your users to a port that isn’t listening externally. The ACE can look for these redirects and rewrite them to include the https://

Here’s how:

action-list type modify http REWRITE
   ssl url rewrite location "website\.domain\.com"

This will take any redirects sent from your server on http://website.domain.com and change them to https://website.domain.com.

Once you have your rewrite command written, all you have to do is assign it to a class-map.

class MyClass
   serverfarm MyServerfarm
   action REWRITE
URL Rewrite for SSL Termination