Debug SSL Handshake Failures (F5, *nix)

This article primarily applies to debugging SSL handshake failures on F5 LTM, but it can be used on any device with tcpdump. 


It can be tricky to truly understand who is affected when you change settings on your F5 SSL profiles. F5 has a handy little counter under the Statistics tab for your virtual-server, but it doesn’t tell you anything about who is failing.


They also log SSL handshake errors (01260009), but again, that doesn’t tell you who is failing.

Let’s say our security team asked us to change the F5’s ciphers, TLS or some other setting. Who did I break? Unless you have a way to talk to the customer, look at a DB for less data, etc. this can be tricky.

TCPDUMP and Wireshark can give us some insight into this, with the right capture.

The foundation for this was a response found here. However, I wanted to take a look at only the handshake failures in Wireshark to get an idea of the customer IP’s that are affected.

If I run a basic capture on the interface where SSL traffic terminates, I can see messages like this:


The actual content we are looking for always starts with 0x15 in hex.


Using the foundation article above, we can craft a tcpdump command to look for these messages.

tcpdump -ni public -C 100 -W 5 -w /var/tmp/ssl_traffic.pcap "port 443 and (tcp[((tcp[12] & 0xf0) >> 2)] = 0x15)"

This command will create 5 100MB files that will cyclically rotate and overwrite each other for you to analyze. They will mostly contain only the handshake failure messages we are looking for.

Filter Only Handshake Failure Packets

If you want to view statistics only for the ‘Handshake Failures’, take a look at the highlighted hex above. We can apply that as a filter so we only see those packets, and view the statistics on those (described below).

Use the following filter to view only the Handshake Failure packets.

frame contains 15:03:01:00:02:02:28

Now the IP’s that are failing to establish an SSL handshake can be analyzed. In Wireshark, using the Statistics tab, click Endpoints. Sort by Packets to see who the top offenders are. This can be used by others to determine if they are legit or not.




Maybe you want to see what ciphers/protocol the client proposed before they failed to analyze further?

Well–add an or statement to our tcpdump statement, you will see both that info. Expand the Client Hello in wireshark, and check what they are proposing. Perhaps that will help you to determine what ciphers you minimally need.

tcpdump -ni public -w /var/tmp/ssl_traffic.pcap "port 443 and ((tcp[((tcp[12] & 0xf0) >> 2)] = 0x15) or (tcp[((tcp[12] & 0xf0) >> 2)] = 0x16))"
Debug SSL Handshake Failures (F5, *nix)

Configure OCSP Stapling – F5 LTM

The documentation that F5 provides for configuring OCSP stapling is pretty sparse. I decided to write up this quick tutorial to supplement their documentation. What is presented below worked for me in my environment, but may not work in all.

Configure a DNS Resolver

  1. Click Network > DNS Resolvers
  2. Click Create… on the right side
  3. Name it whatever you wish, I called mine resolver.
  4. Leave the rest of the settings as the default.ocsp1
  5. Click Finished

Configure Forward Zones

  1. Once your resolver is created, click its name from the Resolver List page.
  2. Click Forward Zones on the top tab area
  3. Click Add…
  4. In the name area put a period “.”
  5. Place the DNS servers that your F5 will use for lookupsocsp2
  6. Click Finished

Import Certificate Chain

Depending on where the certificate was purchased used by your virtual-server, a chain must be created that matches the server-certificate.

Collect the intermediate certificate(s) related to your certificate from the CA. This will contain the URL the F5 will use to validate the certificate using OCSP.

  1. Click File Management > SSL Certificate List
  2. Click Import…
  3. I like to take the chain and paste it in as text, but you can import the chain however you like.ocsp3
  4. Click Import

Create OCSP Profile

  1. Click Profiles > SSL > OCSP Stapling
  2. Click Add…
  3. Click the Advanced drop-down
  4. Pick a name that makes sense to you
  5. In the DNS Resolver section pick the resolver we made above
  6. Under Trusted CA and Trusted Responders pick the chain we created above
  7. I use Comodo certificates, so the settings beginning at Sign Hash and below that are highlighted in yellow were changed to match Comodo’s and F5’s recommendations. These may or may not work for you.ocsp4

Create SSL Profile with OCSP Responder

  1. Go to Local Traffic > Profiles > SSL > Client
  2. Click Create…
  3. Name it whatever makes sense to you
  4. Pick the Certificate, Key and Chain that you have imported already.
  5. In the OCSP Stapling Parameters pick the profile we created in the previous step
  6. Click Add for each certificate the profile will provide.
    • Optional – To create a more secure profile:
      1. Change your Ciphers to; ECDHE+AES-GCM:ECDHE+AES
      2. Disable Renegotiation
    • Click Finished


There are a few ways to test your profile to see whether OCSP responses are being sent from your virtual-server or not. I prefer to run a capture but you can check using the tool at SSL-labs also.

  1. Capture a SSL handshake between you and the virtual server
    • Go to the packet where the vs responds with the Certificate
    • Drill down to the Certificate Status Record Layer
    • Expand until you see OCSP response with a responseStatus of Successful (0)ocsp5
  2. Run a test using SSL-Labs
    • Go to and enter your domain if it is pubic
    • Wait for the test to complete and check for the OCSP Stapling line and make sure it says Yesocsp6
    • I like this site to check for potential issues with your configuration, save it and use it in the future if you don’t already.

Important Note about Default Route

Initially I had an issue with lookups and the OCSP status check using the OCSP Resolver profile I configured. I use Auto-Last Hop on our F5, so my configuration has no default route.

When you have no default route, the default behavior of the F5 is to perform DNS lookups and pull the OCSP status from the virtual-server(s) VLAN self-IP with the OCSP profile assigned to it. If you are firewalling and don’t have a rule permitting this, you may see that OCSP is not working.

As of this version (11.6.0 HF5), the F5 will not use the MGMT interface which typically has its own IP with a gateway. You must have a self-IP assigned elsewhere and configure a default route for it.

There are a few solutions to this:

  • Add a default route for an interface you want to perform lookups (syslog, snmp interface, etc.) and allow that IP to perform DNS lookups and pull the OCSP status from the Internet. The self IP(s) of the F5 on this VLAN must have Internet access.
  • Add ACL’s to allow each virtual-servers VLAN self-IP to perform DNS lookups and pull the OCSP status. These IPs will all require Internet access.
  • Change the OCSP profile to use proxy to perform lookups and pull status.
Configure OCSP Stapling – F5 LTM

Configure SSL Termination on ACE

There are three methods of SSL proxying on the ACE; SSL Initiation, SSL termination and end-to-end SSL. In this article I am covering SSL termination.

SSL Termination

This method makes the ACE do the heavy lifting. Extra CPU cycles to encrypt/decrypt SSL traffic are offloaded to the ACE. In the tests that I have done in our development environments–it has given us, on average, a 10% overall performance increase vs. doing end-to-end SSL encryption with IIS 7.

In this case, the backend servers (IIS, Apache, etc.) do not have certificates on them. They are running port 80 (or any other cleartext port) behind the ACE. The default gateway will be either the ACE (one-arm mode) or the firewall (bridged mode) for your hosts.

NOTE: Some applications will send redirects out to the web-clients. If these redirects are not intercepted by the ACE and changed to HTTPS, it will redirect your clients to a port that is most likely not open on the firewall. I wrote an article that covers that issue here.

The only caveats with this methodology is that your clients do not have requirements against traffic being clear-text on the local network. No un-encrypted traffic traverses the Internet, but some customers and certifications do not allow any clear-text transmission of data.

What Serverfarms?

Do all your serverfarms need SSL or do just a few need SSL? Take note of what needs SSL at this point. For my example my SSL serverfarm is as follows:

serverfarm host FARM-SSL
 rserver SRVR1
 rserver SRVR2
 rserver SRVR3
 rserver SRVR4

Generate & Import Certificates

I covered how to do this on a previous post. You can find it here.

Create a VIP for SSL Traffic

What IP will you NAT on the firewall for SSL traffic? Pick one now.

class-map match-any CLASS-IP-SSL
  2 match virtual-address tcp eq https

Create SSL Proxy

You created your key, imported your certs and configured your chaingroup already right? If not follow the steps above.

ssl-proxy service SSL-PROXY
  key MYKEY
  cert My-CERT-2012
  chaingroup MYCHAIN
  ssl advanced-options CIPHER

Easy right? I find this easier than Windows once you get the hang of it.

Create Policy to match Serverfarm

Now you want to create a policy that defines what serverfarm(s) are going to be behind the SSL proxy. In our case we only want one. If you had multiple you would simply add the class-maps under this policy.

policy-map type loadbalance first-match POLICY-SSL
  class class-default
    serverfarm FARM-SSL

Put it all together

Write your policy-map that puts everything together that you can assign to an interface.

policy-map multi-match VIP-SSL
  class CLASS-IP-SSL
    loadbalance vip inservice
    loadbalance policy POLICY-SSL
    loadbalance vip icmp-reply

Assign it to an Interface

In my case I have VLAN interfaces for bridged mode. So I assign it to both, one so the firewall can NAT it and one so the clients can hit the VIP.

interface vlan 100
  description Server Side VLAN
  bridge-group 1
  access-group input PERMIT-ALL
  service-policy input VIP-SSL
  no shut 
interface vlan 200
  description FW Side VLAN
  bridge-group 1
  access-group input PERMIT-ALL
  service-policy input VIP-SSL
  no shut 

interface bvi 1
 ip address
 description ACE IP Address
 no shutdown

That’s it! Now you should be able to hit your VIP IP from your server subnet and it should load the site with SSL. If it doesn’t, check your serverfarms, gateways, listening ports on the hosts, etc.

Configure SSL Termination on ACE

Easiest way to request an ACE Certificate

There are a variety of different ways to generate a CSR, request a certificate and import that cert into the ACE. I’ve found the easiest way to do this is through the console. No need to set up an FTP/TFTP server, just copy and paste. This is a pre-requisite to SSL-proxying on the ACE.

  • Create your CSR parameters.

CSR parameters are what will be pre-populated in your certificate request. The most important of these is the “common-name” which will be the domain your certificate is assigned to.

  1. Define csr-params 
crypto csr-params MY-PARAM
  country US
  state NY
  locality City
  organization-name Acme
  serial-number 1000

There are other csr-parameters you can define if needed:

switch/Context1(config-csr-params)# ?
Configure CSR parameters:
common-name  Configure organization's common name (required parameter)
country      Configure country code (required parameter)
email        Configure email address
locality     Configure locality name
organization-name  Configure organization name
organization-unit  Configure organization unit's name
serial-number      Configure serial number
state              Configure state name (required parameter)

2. Define cipher set

Your cipher set defines the level of encryption for your site. Some clients require a specific level of encryption, while others do not. Perhaps you need to comply with some business certification (SAS70, SOX, etc.) that controls the overall level of your ciphers. Check with your business or clients for what to define this as.

The session-cache timeout is pretty self explanatory. Some users want it longer than the default. It’s good to define this so when users ask you can always check the config.

parameter-map type ssl CIPHER
  cipher RSA_WITH_RC4_128_SHA
  cipher RSA_WITH_AES_128_CBC_SHA
  cipher RSA_WITH_AES_256_CBC_SHA
  session-cache timeout 600

3. Generate your RSA key pair.

switch/Context1# crypto generate key 2048 MYKEY

I always choose a 2048 bit key because many certificate authorities require at least a 2048 bit key. For many CA’s, 1024 used to be the minimum but many have moved up to 2048.

4. Generate your CSR.

switch/Context1# crypto generate csr MY-PARAM MYKEY

You will be presented with the CSR. If you are using putty this is easy. Just select everything from BEGIN to END and paste into a plain-text file. Name it MYCSR.csr or whatever relates to what you are requesting.

5. Submit your CSR to a CA (godaddy, comodo, etc.)

6. Once your certificate request has been approved you should get an email or ZIP file containing three things:

  • Root certificate
  • Intermediate certificate
  • Your certificate

7. Import root, intermediate and your certificate. Again the easiest way to do this is in putty.

  • Open your root certificate in notepad. Copy it.
switch/Context1# crypto import terminal ROOT-CERT

Paste the cert.

  • Open your intermediate certificate in notepad. Copy it.
switch/Context1# crypto import terminal INTERMEDIATE-CERT

Paste the cert.

  • Open your server certificate in notepad. Copy it.
switch/Context1# crypto import terminal My-CERT-20xx

Paste the cert.

8. Now that all your certificates are imported, you’ll want to create a chaingroup with root and intermediate certs.

crypto chaingroup MYCHAIN
  cert ROOT-CERT

9. Verify that your certificates are stored on the ACE correctly.

switch/Context1# sh crypto files
Filename              File File Expor Key/
                      Size Type table Cert
MYKEY                 1679 PEM  Yes   KEY
ROOT-CA-CERT          1740 PEM  Yes   CERT
My-CERT-2012          2045 PEM  Yes   CERT
cisco-sample-cert     1082 PEM  Yes   CERT
cisco-sample-key      887  PEM  Yes   KEY

Now your certificates are on the ACE! All you need to do now is set up a proxy service.

Easiest way to request an ACE Certificate

Create a Software Load Balancer w/ Content Switching and SSL

I was posed with a problem at work. We have a hardware load-balancer in our production environment (Cisco ACE). We were required to build out a disaster recovery site without the funds to purchase a hardware load-balancer. I then began my research to make it happen. I had a really hard time finding a document for this that was end-to-end, so I decided to write this.

This was originally created back in 2011 so some of the software versions may be a little behind. You can try newer versions if you like, the process should be fairly similar.

The load-balancer software is called HAproxy and it runs on Linux systems. You can install it on any distribution that you like, but for this guide I used Ubuntu Server 9.10. I cant guarantee the same results from a different distro.

Install the OS

  1. Download Ubuntu server 9.10 and install it as a VM or a physical server.
  2. Make sure you have correct IP settings so the server can access the internet, or else apt will not be able to install properly.
  3. When the software package installation screen comes up, only install OpenSSH server.
  4. Log in as the user you specified at installation.
  5. Enter the su -i command to gain privileges to reset the root password
  6. Enter passwd root to change the password to something specific.

Install HAProxy

  1. Once you get your OS up and running you need to install HAproxy.
  2. Install HAproxy with apt. Run the following command (you have to be logged in as root or use su -i)
  • apt-get install haproxy
  1. Back up the old config file
    • cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxyORIGINAL.cfg
  2. Paste the following into your config file and build off of it (this contains content switching)
   log local0
   maxconn 4096
   uid 99
   gid 99

   mode http
   log global
   option tcplog
   option httpclose
   retries 3
   maxconn 2000
   contimeout      5000
   clitimeout      50000
   srvtimeout      50000

frontend LB1 *:80
   option forwardfor
   reqadd          X-Forwarded-Proto:\ https
   reqadd          FRONT_END_HTTPS:\ on
   acl FARM1-acl url_sub -i Hello
   acl FARM2-acl url_sub -i Goodbye
   use_backend Hello if FARM1-acl
   use_backend Goodbye if FARM2-acl
   default_backend DEFAULT

backend FARM1
   stats enable
   stats auth admin:password10
   balance roundrobin
   #option ssl-hello-chk
   server SERVER1 check
   server SERVER2 check

backend FARM2
   stats enable
   stats auth admin:password10
   balance roundrobin
   #option ssl-hello-chk
   server SERVER3 check
   server SERVER4 check

backend DEFAULT
   stats enable
   stats auth admin:password10
   balance roundrobin
   #option ssl-hello-chk
   server SERVER5 check
   server SERVER6 check
  1. Now set HAProxy to start on reboot.
    • vim /etc/default/haproxy
    • Change the ENABLED = 0 to 1
  2. Restart the service and make sure you can hit your frontend IP on port 80.
    • service haproxy restart
  3. When you make changes to the /etc/haproxy.cfg file make sure to run service haproxy restart

Install Stunnel

Stunnel handles the SSL backend connections. Without Stunnel, content switching would not be possible as the traffic would be encrypted.
  1. Create a directory to download stunnel and it’s patch to
    • mkdir /root/stunnel
    • cd /root/stunnel
  2. Download Stunnel and it’s patch (in this version i used version 4.32)
  3. Patch Stunnel
    • cd stunnel-4.32
    • patch -p1 < ../stunnel-4.32-xforwarded-for.diff
  4. Install prerequisites
    • apt-get install libcurl3-openssl-dev
  5. Install stunnel
    • ./configure
    • make && make install

Generate Self-signed Certs

This step is for TESTING ONLY. Do this if if you dont have your real certs yet. To verify everything you’ll need a certificate and private key.
  1. Install OpenSSL:
    • apt-get install openssl
  2. Make a certificate directory:
    • mkdir /etc/certs
    • cd /etc/certs
  3. Generate Private key:
    • openssl genrsa -out server.key 1024
  4. Generate CSR:
    • openssl req -new -key server.key -out server.csr
  5. Generate Self-signed cert
    • openssl x509 -req -days 1200 -in server.csr -signkey server.key -out server.crt
  6. Make the stunnel config file
    • mkdir /etc/stunnel
    • vim /etc/stunnel/stunnel.conf
  7. Add the following to the basic config for stunnel:
key = /etc/certs/server.key
;setuid = nobody
;setgid = nogroup

pid = /etc/stunnel/
debug = 3
output = /etc/stunnel/stunnel.log


  1. Set stunnel to start on reboot
    • vim /etc/init.d/stunnel
    • Paste the following into that file:
# stunnel      This shell script takes care of starting and stopping
#              stunnel
# chkconfig: 345 80 30
# description:  Secure tunnel

# processname: stunnel
# config: /etc/stunnel/stunnel.conf
# pidfile: /var/run/stunnel/

# Source function library.
. /lib/lsb/init-functions

# Source stunnel configureation.
if [ -f /etc/sysconfig/stunnel ] ; then
 . /etc/sysconfig/stunnel


start() {
 # Start daemons.

 echo -n $"Starting $prog: "
 if test -x /usr/local/bin/stunnel ; then
   /usr/local/bin/stunnel /etc/stunnel/stunnel.conf
 [ $RETVAL -eq 0 ] && touch /var/lock/stunnel
 return $RETVAL

stop() {
 # Stop daemons.
 echo -n $"Shutting down $prog: "
 killproc stunnel
 [ $RETVAL -eq 0 ] && rm -f /var/lock/stunnel
 return $RETVAL

# See how we were called.
case "$1" in
 if [ -f /var/lock/stunnel ]; then
 status stunnel
 echo $"Usage: $0 {start|stop|restart|condrestart|status}"
 exit 1

exit $RETVAL
  1. Save it and set it so it can be executed
    • chmod 755 /etc/init.d/stunnel
  2. Test the service
    • Service stunnel stop
    • Service stunnel start
    • (you might need to run the start command a couple times before it works)
  3. Install chkconfig so the service can start on it’s own
    • apt-get install chkconfig
    • chkconfig stunnel on
  4. Restart the server for good measure and make sure your services start.
  5. Finally test the function of HAproxy by going to http://host and https://host with the respective URL paths to test content switching.

Export Real Certificate

When you are done testing you will want to use the actual certificate for the servers that you are load-balancing for. In my case the real certificate was on a windows box. I had to export the certificate and private key from IIS and import it into something stunnel could understand.
  1. Go to the IIS server that has the certificate.
  2. Browse to the site that has the cert and right click it then go to properties.
  3. Click the Directory Security tab and click View Certificate…
  4. Click Details → Copy to File…
  5. Click Yes, export the private key
  6. Uncheck enable strong protection and click next. Make sure all sub boxes are not checked.
  7. Leave the password fields blank
  8. Pick the path where you want to save it.
  9. Click Next → Finish
  10. Take that file and copy it to your load balancer
  11. Once it is copied over cd to the directory that you moved it to and run the following command:
    • openssl pkcs12 -in mycertificate.pfx -nodes -out CERT.pem
  12. Generate your private key associated with the cert.
    • openssl rsa CERT.pem -out mycertificate.key
  13. Edit your /etc/stunnel.conf file to reflect the changes. E.g.
    • cert=/etc/certs/CERT.pem
  14. Restart stunnel

Enable Logging

  1. Install syslog if it is not already installed
    • apt-get install syslogd
  2. Change the config of syslog to allow socket connections
    • vim /etc/default/syslogd
    • SYSLOGD=”-r”
  3. Set the path to log to
    • vim /etc/syslog.conf
    • local0.* /var/log/messages
  4. Restart syslog
    • /etc/init.d/sysklogd restart
  5. Make sure haproxy is set up to log
  log local0

  log global
  option tcplog

 For Servers that have Backend SSL’s

Use nginx. The load-balancing algorithm isnt as effictive as HAProxy, but HAProxy does not support servers on the backend with SSL certificates. This example uses the same base version of ubuntu server with SSH server installed only.
  1. Install nginx
    • apt-get install nginx
    • Install from source:
./configure  --conf-path=/etc/nginx/nginx.conf --with-http_dav_module  --with-http_ssl_module  --with-http_stub_status_module  --add-module=/root/http-healthcheck  --add-module=/root/upstream

aptitude show nginx
apt-get install libc6
apt-get install libgd2-noxpm
apt-get install libgd2-xpm libgeoip1 libpcre3 libssl0.9.8 libxml2 libxslt1.1 zlib1g lsb-base
  1. Install chkconfig
    • apt-get install chkconfig
  2. See if nginx starts with the default config
    • service nginx restart
  3. Make sure you have your SSL certificate and key ready. Without it the next steps arent going to work. Refer the above section on “Export Real SSL
  4. Modify the default nginx config. This example uses content filtering with front-end SSL proxying.
user www-data;
worker_processes  2; #this server has two processors
worker_cpu_affinity 01 10; #assigning each worker to a processor

error_log  /var/log/nginx/error.log;
pid        /var/run/;

events {
    worker_connections  1024;

http {
    include       /etc/nginx/mime.types;

    access_log  /var/log/nginx/access.log;

    sendfile        on;

    keepalive_timeout  65;
    tcp_nodelay        on;

    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;

#defining the farms and backend ports
upstream DEFAULT {

upstream FARM1 {

upstream FARM2 {

#define the server
server {


  ssl on;
  ssl_certificate /etc/certs/CERT.pem; #location of cert
  ssl_certificate_key /etc/certs/KEY.key; #location of key


#location is where content filtering happens. Each path after the / will get directed to the 
#farm defined below w/ the proxy_pass command
  location ~* ^/virtual1|virtual2|virtual3 {

   proxy_set_header  X-Real-IP  $remote_addr;

   proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header Host $http_host;
   proxy_redirect off;
   proxy_max_temp_file_size 0;

   proxy_pass https://VIRTUAL1;


  location ~* ^/virtual4|virtual5 {

   proxy_set_header  X-Real-IP  $remote_addr;

   proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header Host $http_host;
   proxy_redirect off;
   proxy_max_temp_file_size 0;

   proxy_pass https://VIRTUAL5;


  location ~* ^/virtual6|virtual7 {

   proxy_set_header  X-Real-IP  $remote_addr;

   proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header Host $http_host;
   proxy_redirect off;
   proxy_max_temp_file_size 0;

   proxy_pass https://VIRTUAL6;



  1. Restart nginx
    • service nginx restart
  2. Set nginx to start on reboot
    • chkconfig nginx on
  3. Make sure nginx is running properly with the netstat -lp command
root@LBHOST:~# netstat -lp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 LBHOST.local:https *:*                     LISTEN      1322/nginx
tcp        0      0 *:www                   *:*                     LISTEN      1322/nginx
tcp        0      0 *:ssh                   *:*                     LISTEN      882/sshd
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN      882/sshd
Create a Software Load Balancer w/ Content Switching and SSL