Install CA Certificate on F5 Configuration Utility (Management) Interface


It seems that installing a CA signed certificate on the configuration utility (CU) is not a common practice for customers using F5 devices. I have to admit I’ve never had a need myself, until it was required by a security audit.

I followed the available articles, K42531434  contains a mixed bag of information, referencing how to configure the F5 itself as a CA and/or importing a CA cert to the CU through the command line. Not quite what I was looking for. K14620 has some more information, but again, didn’t really apply to what I needed to do.

I did what seemed obvious and generated CSR, got a cert, and applied it. But the problem was the F5 CU does not send any intermediate chain certificates to the client when they connect. Depending on what your browser/OS has in its store, this can cause trust issues, which it did for me. I needed the F5 to send the intermediate certs along with the server cert.

So I opened a case with F5, after the support tech did some digging, he provided me with some information that I’ll cover below.

These steps have been tested with a device running 13.1, it may or may not be the same for 11-12.x.

Generate CSR

1. Log in to the F5 CU.

2. Go to System > Certificate Management > Device Certificate Management > Device Certificate. The default self-signed cert will be listed.


3. Click Renew…


4. Change the drop-down from Self to Certificate Authority and enter the information you need.


5. Click Finished

6. Copy or download the CSR to provide to your certificate authority.


7. Click Finished

Submit the CSR to the certificate authority you use. When the certificate is provided capture the server certificate and intermediate certs to use later.

Import Certificate

1. Go to System > Certificate Management > Device Certificate Management > Device Certificate

2. Click Import…

3. Paste or Upload only the server certificate provided by the CA.


4. At this point, the certificate listed under Device Certificate should be the CA server cert. Test by connecting with a browser to the CU, or openssl to the CU interface.

# openssl s_client -showcerts -connect x.x.x.x:443

Import Intermediate Certificates

This is the part that I couldn’t find documented by F5 until I opened a support case.

1. Copy the intermediate certificate(s) to your clipboard.

2. SSH to the F5 advanced shell

3. Create a new file and paste to:


4. Set security of file:

# chmod 0644 /config/httpd/conf/ssl.crt/intermediate_ca.crt

Configure the F5 to Serve the chain

1. Enter the following command:

# tmsh modify / sys httpd ssl-certchainfile /etc/httpd/conf/ssl.crt/intermediate_ca.crt

2. Restart httpd:

# bigstart restart httpd

3. Test using a browser (using a new session) or with openssl on the advanced shell:

# openssl s_client -showcerts -connect x.x.x.x:443

4. If that works, save the config.

# tmsh save sys config


Install CA Certificate on F5 Configuration Utility (Management) Interface

Debug SSL Handshake Failures (F5, *nix)

This article primarily applies to debugging SSL handshake failures on F5 LTM, but it can be used on any device with tcpdump. 


It can be tricky to truly understand who is affected when you change settings on your F5 SSL profiles. F5 has a handy little counter under the Statistics tab for your virtual-server, but it doesn’t tell you anything about who is failing.


They also log SSL handshake errors (01260009), but again, that doesn’t tell you who is failing.

Let’s say our security team asked us to change the F5’s ciphers, TLS or some other setting. Who did I break? Unless you have a way to talk to the customer, look at a DB for less data, etc. this can be tricky.

TCPDUMP and Wireshark can give us some insight into this, with the right capture.

The foundation for this was a response found here. However, I wanted to take a look at only the handshake failures in Wireshark to get an idea of the customer IP’s that are affected.

If I run a basic capture on the interface where SSL traffic terminates, I can see messages like this:


The actual content we are looking for always starts with 0x15 in hex.


Using the foundation article above, we can craft a tcpdump command to look for these messages.

tcpdump -ni public -C 100 -W 5 -w /var/tmp/ssl_traffic.pcap "port 443 and (tcp[((tcp[12] & 0xf0) >> 2)] = 0x15)"

This command will create 5 100MB files that will cyclically rotate and overwrite each other for you to analyze. They will mostly contain only the handshake failure messages we are looking for.

Filter Only Handshake Failure Packets

If you want to view statistics only for the ‘Handshake Failures’, take a look at the highlighted hex above. We can apply that as a filter so we only see those packets, and view the statistics on those (described below).

Use the following filter to view only the Handshake Failure packets.

frame contains 15:03:01:00:02:02:28

Now the IP’s that are failing to establish an SSL handshake can be analyzed. In Wireshark, using the Statistics tab, click Endpoints. Sort by Packets to see who the top offenders are. This can be used by others to determine if they are legit or not.




Maybe you want to see what ciphers/protocol the client proposed before they failed to analyze further?

Well–add an or statement to our tcpdump statement, you will see both that info. Expand the Client Hello in wireshark, and check what they are proposing. Perhaps that will help you to determine what ciphers you minimally need.

tcpdump -ni public -w /var/tmp/ssl_traffic.pcap "port 443 and ((tcp[((tcp[12] & 0xf0) >> 2)] = 0x15) or (tcp[((tcp[12] & 0xf0) >> 2)] = 0x16))"
Debug SSL Handshake Failures (F5, *nix)

Automate F5 Backups


I like to use Solarwinds CatTools to back up my network gear. It’s a great tool that allows you to easily schedule automated backups of network devices–or any command-line device for that matter.

This is where this post come in. There is no inherent template for backing up an F5 device using CatTools (at least right now).

I have a template that I’ve used over the years to create a UCS file based on the hostname and date, store a local copy in the event I need to restore it, as well as TFTP the file to the built-in TFTP server in CatTools for a remote backup.

CatTools Config:

First you’ll need a copy of CatTools somewhere that can reach your F5 via. SSH and have the ability to TFTP from the F5 to the CatTools server. I won’t go over the install or making sure your ACL’s allow this, but make sure you do this first.

  1. Open CatTools Manager
  2. Add your F5 devices under the Devices tab by clicking Add
  3. Select F5 as the vendor and F5.BigIP as the device type (although I don’t think this ultimately matters for much other than reporting).
  4. Pick a name and type in a Host Address (IP address of the F5 management or self IP with ssh permitted inbound)
  5. Pick SSH2 as your method. f5bkp1
  6. Click the Passwords tab, and add a user allocated just to CatTools. It will need to be an administrator with advanced shell. f5bkp2

Now you have your device(s) ready for backup. I like to make sure I have L3/4 connectivity by testing the Telnet/SSH button to make sure a prompt comes up. If not, work on your routing/ACL’s.

  1. Click the Add button under the Activities tab
  2. Under Activity, pick Device.CLI.Send commands option
  3. Add a description that makes sense to you. f5bkp3
  4. Click the Time tab, and set your schedule however you’d like.
  5. Under Devices, pick the F5(s) you added in the first steps.

The Script

Under the Options tab is where the commands will be entered on the device. The script has several lines which I’ll describe below.

export date=`date +"%y%m%d"​`

This sets an environment variable to the current date, year, month and day for file naming. Change this depending on your location or needs.

export filename=$HOSTNAME.$date.ucs

Now we set the filename to the hostname of the F5 and the date we just defined, followed by ucs.

tmsh save /sys ucs /var/local/ucs/$filename

Now we execute the tmsh command to create and save the UCS file to the directory we want.

NOTE: Depending on how frequently you set the backup, you could fill up your F5’s local storage. Although this would take a long time, you can create a script to auto-delete the files if you want, or delete them every 6 months or so when you are in the UI.

cd /var/local/ucs
tftp -m binary -c put $filename

Now we move to that directory, and TFTP the file to the Cattools server. Change the 192 address to your Cattools server.

This is what it will look like in the UI:


Click OK and run the activity. If you did everything right there should be UCS files in your TFTP directory in CatTools. If not, check ACL’s or routing to make sure you aren’t missing anything.


Automate F5 Backups

F5 HTTP Response-Code Alerts


UPDATE: Added more logic to remove reporting anomalies from the F5 device (inaccurate response code values). The IF statement before the email pipe will cause the report to not send if the value is empty or the top server responded with less than 15% of the total response codes of that type.

I love the F5 analytics feature, specifically for HTTP. The cool thing about the feature, is that you can get a very good idea of an issue based on response code behavior.

Let’s say you have a URL that is monitored by an external HTTP tool. This tool checks the URL to make sure it gets a 200 HTTP response-code back every 60 seconds. I just got an email that the check failed, but this URL has 10 nodes behind it. How do I know which node is failing? 

Well, you can manually log into the F5 and check through the UI, use the Rest API or some other utility that ties into the F5 API.

  • Manually checking is not ideal, we want automation.
  • The Rest API is great, but why invest a ton of time getting that working, if this is our only goal?

My preference would be to add this to the Analytics Profile on the F5, so any administrator could see it and know what’s happening. Unfortunately, F5 does not alert on HTTP response-code in the Analytics Profile (as of 12.1).

Hopefully F5 does this at some point, but until then, I’m going to show you how to do this with a bash script on the F5 device itself.


  1. Determine which response code do you want to check for
  2. Determine how often you want to check for potential issues
  3. Make sure you have an email server that the F5 can forward the message through.
  4. Make sure you have access to the F5 advanced shell

The Script

I’ve done most of the heavy lifting for you. I intentionally defined any part that I thought may need to be customized based on usage as variables, change them to suit your needs.

Place it wherever you want, probably somewhere under /root is best so someone could find it easily. Before you schedule the script, make sure you chmod 755 it so it can be executed (chmod 755 /root/


#** Define your environment **
#** specific variables here **

#Which response code do you want to look for?
#Add a trailing space to prevent any possible time matches 
#(the year is followed by : in the output)
code="302 "

#How many minutes in the past do you want your script to look?
#Match this up with cron

#Topx determines when an email is triggered. If you are implementing this 
#in a dev environment with not much going on, you may want to limit your 
#response code trigger to when it is in the top 4 of all response codes. 
#If you are running in a prod environment, maybe top 7 is fine. 
#If you are getting too many emails, make the number lower. If you want to 
#make sure things are working, make the number higher.

#Email configuration
subject="$code Error Notification"


# Determine number of response codes in the past x minutes. 
# Typically if response codes are in the topx response-code 
# types, something we care about is happening. 

numcode=(`tmsh show analytics report view-by response-code limit ${topx} range now-${howoften}m | grep $code | awk '{print $3}'`)

# Start our logic to send alerts
# Change the number to insert more granularity than your alerting
if [[ "$numcode" -gt 11000 ]]; then
 # What is the hostname of this F5 for the email
 hostname=`uname -n`
 # This will break down the top culprit of the
 # queried response code into three values in an array
 badguy=(`tmsh show analytics http report view-by pool-member drilldown { { entity response-code values { ${code} } } } limit 1 range now-${howoften}m| tail -n+8 | sed 's/:/ /;s/ | / /'`)
 badguysname=`nslookup ${badguy[0]} | awk -F "name = " '{print $2}' | sed '/^\s*$/d'`
 if [[ -n "$badguy" && "$((${badguy[2]}*100/$numcode))" -gt 15 ]]; then
 ( echo "${code}Response Code Report";
 echo "------------------------------";
 echo "";
 echo "Overall, there have been *$numcode*, $code messages in the past $howoften minutes for all VIPs running on $hostname.";
 echo "";
 echo "The top offender of these messages is ${badguy[0]} on port ${badguy[1]}.";
 echo "";
 echo "This server has generated $((${badguy[2]}*100/$numcode))% of the total ${code}messages in this time window.";
 echo "";
 echo "nslookup tells us the bad server is: $badguysname ";
 echo "";
 echo "Total $code : $numcode"
 echo "Node $code : ${badguy[2]}"
 echo "Bad Node : $badguysname"; ) | mailx -S smtp="$smtpserver" -r "$sender" -s "$subject" -v "$receiver" 
 echo "${code}response codes are not in the top $topx overall for this device. Script ended, no email sent."

The email portion is a bit clunky, but with the version of software I was running, I wasn’t able to send an HTML email. If you figure that out, or want to edit the email content, go for it.

Scheduling the Script

Now we need to schedule the script. Use crontab -e to schedule your script. Read this article from F5 for some tips if you need more info. Make sure you match up the timing of your cron task with the script defined time window.

Make sure you document that you have done this!! If you leave your job and a new administrator comes in, they would have to do some digging to figure out what you have done.

I would recommend testing this by setting your response code to 200, if you get emails based on the schedule you set, change to what you want to look for.

F5 HTTP Response-Code Alerts

Configure OCSP Stapling – F5 LTM

The documentation that F5 provides for configuring OCSP stapling is pretty sparse. I decided to write up this quick tutorial to supplement their documentation. What is presented below worked for me in my environment, but may not work in all.

Configure a DNS Resolver

  1. Click Network > DNS Resolvers
  2. Click Create… on the right side
  3. Name it whatever you wish, I called mine resolver.
  4. Leave the rest of the settings as the default.ocsp1
  5. Click Finished

Configure Forward Zones

  1. Once your resolver is created, click its name from the Resolver List page.
  2. Click Forward Zones on the top tab area
  3. Click Add…
  4. In the name area put a period “.”
  5. Place the DNS servers that your F5 will use for lookupsocsp2
  6. Click Finished

Import Certificate Chain

Depending on where the certificate was purchased used by your virtual-server, a chain must be created that matches the server-certificate.

Collect the intermediate certificate(s) related to your certificate from the CA. This will contain the URL the F5 will use to validate the certificate using OCSP.

  1. Click File Management > SSL Certificate List
  2. Click Import…
  3. I like to take the chain and paste it in as text, but you can import the chain however you like.ocsp3
  4. Click Import

Create OCSP Profile

  1. Click Profiles > SSL > OCSP Stapling
  2. Click Add…
  3. Click the Advanced drop-down
  4. Pick a name that makes sense to you
  5. In the DNS Resolver section pick the resolver we made above
  6. Under Trusted CA and Trusted Responders pick the chain we created above
  7. I use Comodo certificates, so the settings beginning at Sign Hash and below that are highlighted in yellow were changed to match Comodo’s and F5’s recommendations. These may or may not work for you.

NOTICE!! When I wrote this article (and currently), the F5 article infers to set ‘Status Age’ to 86400 as the image below reflects. After upgrading to 13.x, OCSP stapling broke in my environment. A case with F5 determined that this should be set to zero. In summary:

“Status Age specifies the maximum allowed lag time that the BIG-IP system accepts for the ‘thisUpdate’ time in the OCSP response. If this maximum is exceeded, the response is dropped. If this value is set to 0 (zero), this validation is skipped. The default value is 0 seconds. In a nutshell, this means that the BIG-IP will silently discard a response *that carries a nextUpdate time* when thisUpdate is more than or has exceeded status-age seconds in the past.”


Create SSL Profile with OCSP Responder

  1. Go to Local Traffic > Profiles > SSL > Client
  2. Click Create…
  3. Name it whatever makes sense to you
  4. Pick the Certificate, Key and Chain that you have imported already.
  5. In the OCSP Stapling Parameters pick the profile we created in the previous step
  6. Click Add for each certificate the profile will provide.
    • Optional – To create a more secure profile:
      1. Change your Ciphers to; ECDHE+AES-GCM:ECDHE+AES
      2. Disable Renegotiation
    • Click Finished


There are a few ways to test your profile to see whether OCSP responses are being sent from your virtual-server or not. I prefer to run a capture but you can check using the tool at SSL-labs also.

  1. Capture a SSL handshake between you and the virtual server
    • Go to the packet where the vs responds with the Certificate
    • Drill down to the Certificate Status Record Layer
    • Expand until you see OCSP response with a responseStatus of Successful (0)ocsp5
  2. Run a test using SSL-Labs
    • Go to and enter your domain if it is pubic
    • Wait for the test to complete and check for the OCSP Stapling line and make sure it says Yesocsp6
    • I like this site to check for potential issues with your configuration, save it and use it in the future if you don’t already.

Important Note about Default Route

Initially I had an issue with lookups and the OCSP status check using the OCSP Resolver profile I configured. I use Auto-Last Hop on our F5, so my configuration has no default route.

When you have no default route, the default behavior of the F5 is to perform DNS lookups and pull the OCSP status from the virtual-server(s) VLAN self-IP with the OCSP profile assigned to it. If you are firewalling and don’t have a rule permitting this, you may see that OCSP is not working.

As of this version (11.6.0 HF5), the F5 will not use the MGMT interface which typically has its own IP with a gateway. You must have a self-IP assigned elsewhere and configure a default route for it.

There are a few solutions to this:

  • Add a default route for an interface you want to perform lookups (syslog, snmp interface, etc.) and allow that IP to perform DNS lookups and pull the OCSP status from the Internet. The self IP(s) of the F5 on this VLAN must have Internet access.
  • Add ACL’s to allow each virtual-servers VLAN self-IP to perform DNS lookups and pull the OCSP status. These IPs will all require Internet access.
  • Change the OCSP profile to use proxy to perform lookups and pull status.
Configure OCSP Stapling – F5 LTM

F5 Persistence Mirroring w/ iRules


In a failover pair of devices, mirroring persistence records is important. When one LTM fails, we need the other to know which server a user’s session was “stuck” to. If a device failed and the other device was not aware of these records, the decision process would start again and a user could lose their session (wait for the tickets to roll in…) We want our HA pair of F5’s to failover seamlessly so they can be updated without impacting users, as well as transparency to end-users when a device fails.

The problem with the F5 is when you are using multiple persistence methodologies, mirroring does not function for each type you are using in the iRule, only the type in the ‘Default Persistence Profile’ chosen in the virtual-server. This is a problem when you are using a few different types in one iRule. For example; persist cookie, persist source_addr, persist uie, etc. all in one iRule.

So what do I do? The solution to this is difficult if not impossible to find in dev-central. I had to open a support case to even find this out. Use universal persistence!! In almost all cases when I opened support tickets the engineer would state that when I was using ‘persist cookie’ that it was unusual. I always thought to myself, “What, this is crazy!! Why even offer me a persistence type if it is not recommended to be used by the manufacturers of the device themselves!!”

The solution to this problem is to convert your existing iRule that uses a few different types of persistence to all use universal. The syntax may not be as simple, but universal persistence can do everything the other types can and more. Unfortunately F5 doesn’t have universal persistence documented very well (common theme), but this is a good start.

Let’s go over a simple example iRule. In this example I’m examining URI (virtual directory) and making decisions based on that value. I am using source IP and cookie hash stickiness. Different apps require different types of persistence. A normal person would offer what F5 provides, but that is not recommended by F5.

 switch -glob [string tolower [HTTP::uri]] {
 "/app1*" {
 pool POOL-APP1
 persist source_addr
 "/app2*" -
 "/app3*" {
 pool POOL-APP23
 persist cookie hash "ASP.NET_SessionId"
 "/app4*" {
 pool POOL-APP4
 persist none
 default {
 HTTP::redirect https://[getfield [HTTP::host] ":" 1][HTTP::uri]

Unfortunately I can only pick one default persistence type in my virtual-server so not every record will fail over to the second node. We need to fix this by converting the syntax to universal and pick UIE as the default type.

 switch -glob [string tolower [HTTP::uri]] {
 "/app1*" {
 pool POOL-APP1
 persist uie [IP::client_addr]
 "/app2*" -
 "/app3*" {
 pool POOL-APP23
 persist uie [HTTP::cookie "ASP.NET_SessionId"]
 "/app4*" {
 pool POOL-APP4
 persist none
 default {
 HTTP::redirect https://[getfield [HTTP::host] ":" 1][HTTP::uri]

Pretty simple, no? Now I assign my universal persistence profile with the ‘Mirror Persistence’ option checked.


Persistence records will now mirror to the passive device. View the records on each device by running ‘tmsh show ltm persistence persist-records’ to validate that the records are on each F5.

Aside: I think by now you all know how big of a fan I am of the Cisco ACE. Guess how easy this is with the ACE? Add ‘replicate sticky’ to your sticky group configuration. Way too simple…

F5 Persistence Mirroring w/ iRules

Poodle Vulnerability: Who’s Using SSLv3?


There are a ton of different articles on devcentral about certain iRules for changing the encryption that users are using, dropping them, etc. I found it difficult to find any code specific to just logging who is using SSLv3 so I can track them down and try to contact the customer. Managers found the reports that I generated useful to make a business decision instead of simply shutting of SSLv3 on the F5.

All this rule does is define a high-speed-logging server and send a message to the logging server so you can run reports against the logs. It logs source IP and URI (virtual-directory) from the request. Apply this to whichever virtual-server you need to log these requests from.

   set logpub [HSL::open -publisher /Common/HSL-SERVER]
   set cipher [SSL::cipher version]
   set uri [HTTP::uri]
   if { $cipher equals "SSLv3" } {
      HSL::send $logpub "$cipher encryption used from client; [IP::client_addr] to $uri"
Poodle Vulnerability: Who’s Using SSLv3?