Debug SSL Handshake Failures (F5, *nix)

This article primarily applies to debugging SSL handshake failures on F5 LTM, but it can be used on any device with tcpdump. 

handshake

It can be tricky to truly understand who is affected when you change settings on your F5 SSL profiles. F5 has a handy little counter under the Statistics tab for your virtual-server, but it doesn’t tell you anything about who is failing.

sslshake1

They also log SSL handshake errors (01260009), but again, that doesn’t tell you who is failing.

Let’s say our security team asked us to change the F5’s ciphers, TLS or some other setting. Who did I break? Unless you have a way to talk to the customer, look at a DB for less data, etc. this can be tricky.

TCPDUMP and Wireshark can give us some insight into this, with the right capture.

The foundation for this was a response found here. However, I wanted to take a look at only the handshake failures in Wireshark to get an idea of the customer IP’s that are affected.

If I run a basic capture on the interface where SSL traffic terminates, I can see messages like this:

sslshake2

The actual content we are looking for always starts with 0x15 in hex.

sslshake3

Using the foundation article above, we can craft a tcpdump command to look for these messages.

tcpdump -ni public -C 100 -W 5 -w /var/tmp/ssl_traffic.pcap "port 443 and (tcp[((tcp[12] & 0xf0) >> 2)] = 0x15)"

This command will create 5 100MB files that will cyclically rotate and overwrite each other for you to analyze. They will mostly contain only the handshake failure messages we are looking for.

Filter Only Handshake Failure Packets

If you want to view statistics only for the ‘Handshake Failures’, take a look at the highlighted hex above. We can apply that as a filter so we only see those packets, and view the statistics on those (described below).

Use the following filter to view only the Handshake Failure packets.

frame contains 15:03:01:00:02:02:28

Now the IP’s that are failing to establish an SSL handshake can be analyzed. In Wireshark, using the Statistics tab, click Endpoints. Sort by Packets to see who the top offenders are. This can be used by others to determine if they are legit or not.

sslshake4

 

Extra

Maybe you want to see what ciphers/protocol the client proposed before they failed to analyze further?

Well–add an or statement to our tcpdump statement, you will see both that info. Expand the Client Hello in wireshark, and check what they are proposing. Perhaps that will help you to determine what ciphers you minimally need.

tcpdump -ni public -w /var/tmp/ssl_traffic.pcap "port 443 and ((tcp[((tcp[12] & 0xf0) >> 2)] = 0x15) or (tcp[((tcp[12] & 0xf0) >> 2)] = 0x16))"
Advertisements
Debug SSL Handshake Failures (F5, *nix)

Debugging ARP on Cisco ASA

broadcast

The packet capture wizard in ASDM is a great feature of the ASA platform. It allows a network administrator to easily debug an issue and export the capture right to Wireshark from the wizard.

However, as you use this you may notice something. Where are my arp packets? Any time Wireshark is ran from a layer-2 network, arp packets will inevitably be captured. Something I didn’t know is that the ASDM wizard does not capture broadcast packets (at least at the time this was written ASA version 9.4(2) and ASDM 7.6).

Unfortunately Cisco doesn’t really describe this in any of their capture documentation, so if you don’t typically capture through the command line, you’ll never see broadcasts and may wonder what’s wrong.

How can I capture arp broadcasts on my ASA for troubleshooting layer-2 issues?

You have to do this through the ASA command line.

  1. Log in to the ASA you want to capture/see ARP packets.
  2. Use the ‘capture’ command with the ethernet-type arp

An example would be:

ASA# capture arp-cap ethernet-type arp interface inside

Where arp-cap is the name of your capture, the ethernet-type filters the capture to only arp packets and the interface picks the interface where you want to see the broadcasts.

You can define a ‘buffer’ flag if you want, but don’t worry about overloading your ASA, the default is 512kb. The above command is typically what you want.

Now we can execute a show command to see the capture buffer:

ASA# sh cap arp

81 packets captured

1: 13:21:17.283554 arp who-has 192.168.10.1 (cc:3:ca:f8:34:50) tell 192.168.10.21
 2: 13:21:17.283630 arp reply 192.168.10.1 is-at cc:3:ca:f8:34:50
 3: 13:21:18.600005 arp who-has 10.4.49.190 tell 192.168.10.1
 4: 13:21:20.053692 arp who-has 192.168.10.1 (cc:3:ca:f8:34:50) tell 192.168.10.167
 5: 13:21:20.053784 arp reply 192.168.10.1 is-at cc:3:ca:f8:34:50
 6: 13:21:21.069271 arp who-has 10.4.49.182 tell 10.4.48.25
 7: 13:21:21.998391 arp who-has 10.4.49.182 tell 10.4.48.25

We now  see who is broadcasting for what, and what hardware address they reside on. Use the detail flag to see more information.

Clean Up

We’ve got what we need, so its time to clean up. It’s very simple:

ASA# no cap arp-cap
Debugging ARP on Cisco ASA