Debugging ARP on Cisco ASA

broadcast

The packet capture wizard in ASDM is a great feature of the ASA platform. It allows a network administrator to easily debug an issue and export the capture right to Wireshark from the wizard.

However, as you use this you may notice something. Where are my arp packets? Any time Wireshark is ran from a layer-2 network, arp packets will inevitably be captured. Something I didn’t know is that the ASDM wizard does not capture broadcast packets (at least at the time this was written ASA version 9.4(2) and ASDM 7.6).

Unfortunately Cisco doesn’t really describe this in any of their capture documentation, so if you don’t typically capture through the command line, you’ll never see broadcasts and may wonder what’s wrong.

How can I capture arp broadcasts on my ASA for troubleshooting layer-2 issues?

You have to do this through the ASA command line.

  1. Log in to the ASA you want to capture/see ARP packets.
  2. Use the ‘capture’ command with the ethernet-type arp

An example would be:

ASA# capture arp-cap ethernet-type arp interface inside

Where arp-cap is the name of your capture, the ethernet-type filters the capture to only arp packets and the interface picks the interface where you want to see the broadcasts.

You can define a ‘buffer’ flag if you want, but don’t worry about overloading your ASA, the default is 512kb. The above command is typically what you want.

Now we can execute a show command to see the capture buffer:

ASA# sh cap arp

81 packets captured

1: 13:21:17.283554 arp who-has 192.168.10.1 (cc:3:ca:f8:34:50) tell 192.168.10.21
 2: 13:21:17.283630 arp reply 192.168.10.1 is-at cc:3:ca:f8:34:50
 3: 13:21:18.600005 arp who-has 10.4.49.190 tell 192.168.10.1
 4: 13:21:20.053692 arp who-has 192.168.10.1 (cc:3:ca:f8:34:50) tell 192.168.10.167
 5: 13:21:20.053784 arp reply 192.168.10.1 is-at cc:3:ca:f8:34:50
 6: 13:21:21.069271 arp who-has 10.4.49.182 tell 10.4.48.25
 7: 13:21:21.998391 arp who-has 10.4.49.182 tell 10.4.48.25

We now  see who is broadcasting for what, and what hardware address they reside on. Use the detail flag to see more information.

Clean Up

We’ve got what we need, so its time to clean up. It’s very simple:

ASA# no cap arp-cap
Advertisements
Debugging ARP on Cisco ASA

F5 Authentication using Active Directory or LDAP

password-wide

The following instructions will cover how to deploy Active Directory or LDAP authentication with HA (redundant DC’s).

F5 provides a few key articles that build the basis for this summary. Found here, here and here.

Key Information

Local users with the same name as an AD user cannot authenticate with local password once Remote AD authentication is enabled. However, local rights overrule ‘External Users’ configuration.

Example: I have user bsmith local and in AD. bsmith is an admin locally, but ‘External Users‘ is configured as operator only.

Once bsmith authenticates, he will be an administrator on the box.

The built-in ‘admin‘ account is the only local account that will be able to authenticate if AD is down. Make sure you have the password documented.

Prerequisites

  • Name & IP address of LDAP/AD server(s)
  • Distinguished Name of domain
  • Service account for binding to LDAP to monitor servers
  • Optional: Two LDAP/AD servers for HA configuration

Testing

Before starting it’s important to test from the command-line of the F5 to validate network accessibility and the LDAP search string.

What is my distinguished name?

If you’re using Microsoft Active Directory, follow these steps (make sure Advanced Features is on):

1. Open Active Directory Users and Computers

2. Right click the domain name at the top of the tree and click Properties

3. Under attributes you will see ‘distinguishedName‘ the value is important for the rest of these instructions. Capture it.

Pick a test user

A user will be needed to validate during some of our testing. Pick whatever user you’d like, but take note of what OU the user is in. If you are using Microsoft AD it may be in the Users OU.

Build Search String

Merge the information from the DN and user OU to build search string for testing. Here is an example:

Domain: mydomain.com

User OU: users

User: myuser

Execute Test

ldapsearch is the utility used by the F5 for testing. Follow these steps to validate layer-3/4 connectivity and LDAP functionality.

1. SSH to the BigIP

2. Use the following format to test a LDAP query using ldapsearch:

ldapsearch -xLLL -H 'ldap://mydc.domain.com' -b "cn=users,dc=mydomain,dc=com" 
-D mydomain\\domain-user -w 'userspassword' '(samaccountname=myuser)'

Use ‘ldapsearch –help‘ to get more information about the flags.

3. Using the previous query should return a bunch of information about your user. If not, something is wrong with the syntax. Try different variations until you get it working.

If content is returned, we know that the F5 can reach our LDAP server (if it cannot, check that a self-IP exists on the same subnet as the LDAP server or a route exists) and that our DN string is correct for future configuration.

Create LDAP Monitor

A monitor is needed to probe the pool that will be created in the next step. The string created earlier can be used here.

1. Log in to the F5 UI

2. Click Local Traffic > Monitors > Create…

3. Enter a name, i.e. ldap-monitor

4. Under type select LDAP

5. Create an interval that makes sense to you, the defaults are usually fine

6. Under ‘User Name‘ put the user you created in the prerequisites. Ideally it is a service account with no interactive rights, simply used to bind to LDAP.

Important!! If you use a prefix to log on to your domain, i.e. domain\user  you must enter either domain\\user WITH TWO SLASHES or myuser@domain. I’ve had the best luck with the domain\\user format.

7. In the Base field, enter the OU we want to check (bind to).

It can be anything you’d like, but basically we are making sure the LDAP server serves up a response to our LDAP request.

I simply used the cn=users,dc=mydomain,dc=com

8. Under Filter enter the object you want to check. I used cn=Domain Guests since it is a built-in object and it is not used (if I probed Domain Admins the listing could potentially be intercepted and used for nefarious purposes).

9. If your server supports SSL/TLS optionally select one under the Security field.

10. Click Finished

Create LDAP Virtual-Server

This can be done on the local BigIP or a remote device that is accessible by the device LDAP authentication is being implemented.

The purpose of this is so that if an LDAP server fails, the F5 can continue authentication. Without this configuration the F5 must rely on a single server for authentication.

1. Log in to the F5 UI

2. Click Local Traffic > Virtual Servers > Create…

3. Enter any name, IP address (ideally on the same subnet as LDAP servers), Service port is 389

4. Protocol is TCP, with TCP profile

5. Add a Default Pool as a resource with the two domain-controllers in your environment on port 389.

6. Assign the monitor created in the previous step to this pool.

7. Click Finished

8. Create a DNS record for this virtual-server local to your environment, i.e. ldapvip.domain.com

Configure LDAP Authentication

Finally!!

Important Tip: Make sure you have an SSH and browser session already open to your device in-case you get locked out. The default local admin user will always be a fallback in the case this happens, make sure you have those credentials handy.

1. Click System > Users > Authentication > Change

2. Under User Directory select “Remote – Active Directory” or “Remote – LDAP” (I have not experienced any functional difference between these in practice).

3. Host is the DNS record we created in step 8 above. If you skipped the HA portion, just enter the A record for your LDAP server.

4. Remote Directory Tree: This is the OU or starting point for your user container.

Above we used cn=users,dc=domain,dc=com. Also, dc=domain,dc=com could be used, but why return all that content when the Users are only in one or a few OU’s?

If yours are in an OU under that, use the format cn=F5users,cn=users,dc=domain,dc=com

Most cases will be cn=users,dc=domain,dc=com

5. Scope: This determines the level of your search.

Important Tip: I’ve never gotten Base or One to work, only Sub.

6. Bind: Even though this is blue, inferring that it is mandatory, it is not. If you created a service account (we did for our monitor) and you only want this user to be used to bind, then go ahead and enter that user here.

Important Tip: Remember how in the monitor configuration we had to use double slashes? I.e. domain\user. For some reason the same does not apply here. Use only ONE slash. i.e. domain\user

I left the Bind fields empty and used the User Template setting. If a user can’t bind to begin with, why search for that user?

7. Under User template %s@domain.com will attempt to use the user authenticating to bind (the F5 inserts the username typed in the User field for the %s), if they can’t bind (non-existent user) they won’t be looked up.

8. I’m not really sure what ‘Check Member Attribute in Group‘ does functionally (doesn’t Remote Role Groups cover this?), F5’s documentation is lacking here. I leave it unchecked.

9. Under Login LDAP Attribute enter samaccountname

10. In the External Users section assign the Role you’d like for authenticated AD users. Remember that if a local user matches remote, local rights supersede this configuration. Operator is a good choice if this is designed for sys-admins to do what they need.

11. Click Finished

Test logging in from a different browser session!! If all went well, logging in will work. If not something is wrong/missing. Look over the steps above again and make sure nothing was missed.

If not, troubleshoot using tcpdump

Troubleshooting

tcpdump is your best friend for figuring out what’s going on. Luckily LDAP is clear-text so deciphering syntax issues is fairly simple. F5 also has an article covering how to troubleshoot LDAP issues

1. SSH to the F5 performing the LDAP queries.

2. Use the following syntax to run a simple capture

tcpdump -s0 -ni 0.0:nnn -w /shared/tmp/ldapdebug.pcap 'host 10.1.1.1 or host 10.1.1.2'

Replace the 10.x addresses with the IP’s of your LDAP servers.

3. Attempt a login from a browser.

4. Once it fails, stop the capture with ctrl-c

5. SCP the capture off the F5 (will be in the /shared/tmp directory)

6. Examine with Wireshark.

Look for syntax errors after “searchRequest” or “bindRequest” queries. The “bindResponse”, “searchResEntry” or “searchResDone” response will be a good indicator of the problem.

F5 Authentication using Active Directory or LDAP

My F5 Needs HSTS!!

There are plenty of written resources out there about HSTS (HTTP Strict Transport Security). I don’t intend to explain or elaborate on what it is or why you want it. The IETF RFC is a good place to start.

F5 actually has a nice article with an iRule example of HSTS enforcement. HSTS pairs nicely with OCSP Stapling, check my previous article on how to do that on an F5.

However, in my environment we have servers internal to our network that call the same virtual-server as external customers that may send calls using HTTP and HTTPS. We network engineers never really know what developers do or plan on doing, but we don’t want to break anything on them now or in the future.

I’ve added a simple filter so that if calls from the 10.0.0.0/8 network come in to the virtual-server, just pass it on–otherwise, attach the HSTS header. This is optional, so tweak this to apply to your environment. If you don’t care about internal calls, remove the if statement. Once it’s in place, apply to your virtual-server and enjoy!!

UPDATE:

Initially I had RULE_INIT with “1 year” as the clock time determinant. This is good for performance reasons, because the event is only triggered once. Below is from F5’s iRule wiki showing when RULE_INIT is triggered:

hsts0

I noticed this on my iRule statistics page of my iRule, showing that the time that the iRule was saved, was the set time the max-age will be compared against.

hsts1

The problem with this is that over time, your HSTS max-age will slowly shorten unless your F5 is rebooted or the iRule is modified. This is not ideal, since we want this rule to be evergreen.

So, we have a couple options; change RULE_INIT to CLIENT_ACCEPTED or set your clock scan very far into the future to protect yourself. The HSTS draft doesn’t specify that having a very long max-age is not recommended, so at this point if you want to minimize CPU load on your F5, go that route. CLIENT_ACCEPTED will have a performance impact (likely nominal), but you know that your max-age will always be a set period of time, i.e. one year and you’ll never have to come back to this iRule.

CLIENT_ACCEPTED:

when CLIENT_ACCEPTED {
 set static::expires [clock scan "1 year"]
}
when HTTP_RESPONSE {
 if { [IP::client_addr] starts_with 10. } {
 }
 else {
   HTTP::header insert Strict-Transport-Security "max-age=[expr {$static::expires
   - [clock seconds]}]; includeSubDomains"
 }
}

Static Age:

when RULE_INIT {
  set static::expires [clock scan 20900101]
}
when HTTP_RESPONSE {
  if { [IP::client_addr] starts_with 10. } {
}
else {
  HTTP::header insert Strict-Transport-Security "max-age=[expr {$static::expires 
  - [clock seconds]}]; includeSubDomains"
 }
}
My F5 Needs HSTS!!