Debugging ARP on Cisco ASA

broadcast

The packet capture wizard in ASDM is a great feature of the ASA platform. It allows a network administrator to easily debug an issue and export the capture right to Wireshark from the wizard.

However, as you use this you may notice something. Where are my arp packets? Any time Wireshark is ran from a layer-2 network, arp packets will inevitably be captured. Something I didn’t know is that the ASDM wizard does not capture broadcast packets (at least at the time this was written ASA version 9.4(2) and ASDM 7.6).

Unfortunately Cisco doesn’t really describe this in any of their capture documentation, so if you don’t typically capture through the command line, you’ll never see broadcasts and may wonder what’s wrong.

How can I capture arp broadcasts on my ASA for troubleshooting layer-2 issues?

You have to do this through the ASA command line.

  1. Log in to the ASA you want to capture/see ARP packets.
  2. Use the ‘capture’ command with the ethernet-type arp

An example would be:

ASA# capture arp-cap ethernet-type arp interface inside

Where arp-cap is the name of your capture, the ethernet-type filters the capture to only arp packets and the interface picks the interface where you want to see the broadcasts.

You can define a ‘buffer’ flag if you want, but don’t worry about overloading your ASA, the default is 512kb. The above command is typically what you want.

Now we can execute a show command to see the capture buffer:

ASA# sh cap arp

81 packets captured

1: 13:21:17.283554 arp who-has 192.168.10.1 (cc:3:ca:f8:34:50) tell 192.168.10.21
 2: 13:21:17.283630 arp reply 192.168.10.1 is-at cc:3:ca:f8:34:50
 3: 13:21:18.600005 arp who-has 10.4.49.190 tell 192.168.10.1
 4: 13:21:20.053692 arp who-has 192.168.10.1 (cc:3:ca:f8:34:50) tell 192.168.10.167
 5: 13:21:20.053784 arp reply 192.168.10.1 is-at cc:3:ca:f8:34:50
 6: 13:21:21.069271 arp who-has 10.4.49.182 tell 10.4.48.25
 7: 13:21:21.998391 arp who-has 10.4.49.182 tell 10.4.48.25

We now  see who is broadcasting for what, and what hardware address they reside on. Use the detail flag to see more information.

Clean Up

We’ve got what we need, so its time to clean up. It’s very simple:

ASA# no cap arp-cap
Advertisements
Debugging ARP on Cisco ASA

ACL Debugging on ASA with ASDM

I know–that title is a mouthful, but it’s actually pretty simple. A lot of environments out there have ACL’s that have implicit permits because it was simply too much work to get a list of what needs to talk to what before the firewall(s) went into place.

So, you’ve got an ACL applied to an interface that is set to permit all. The goal is to not permit everything and break out individual permit rules so we can change that to a deny rule. In ASDM it should look something like this.

1

So what goes through that ACL that I can start breaking out and writing specific rules for?

Right now if you right click the ACL in ASDM you get this nice little function called “Show Log…” The problem is this doesn’t work right out of the gate. If you click on this option you wont see anything yet.

2

The first thing I like to do is enable Debug logging on ASDM. That way when I open the log viewer or try to see hits on an ACL–I don’t have to modify anything.

Under Configuration –> Logging –> Logging Filters –> ASDM change “Filter on severity” to Debugging

The strange thing is now when you go back to the ACL’s and click “Show Log…” you still don’t see anything. Why? The problem is that permit statements will not log by default. This needs to be enabled either in ASDM or through the command line.

Command Line:

logging message 106100

ASDM:

Configuration > Device Management > Logging > Syslog Setup > 106100 “Disabled = No”

Now when you right click a rule and select “Show Log…” you will start seeing hits on the ACL. Monitor these hits for a while during work hours and you will at least get a huge chunk of the rules written. This is really nice for breaking out permit rules so you can eventually have an implicit Deny All there instead of a Permit All.

ACL Debugging on ASA with ASDM

ASA Appliance to Module Headaches

asa-smheadache

I recently migrated one of our environments from an ASA 55xx appliance to an ASA service module running in a 6509-e chassis. There were a few gotchas that I was not aware of that made the transition date quite an interesting experience… There were three pain-points for me on migration day. I’ll cover what they were and the configuration that was actually needed to make things work.

I use ASDM for most of my day-to-day work on ASA, so I’ll post the configuration with ASDM with the command line equivalent after.

NAT Exempt rules for VPN

I had a heck of a time finding a definitive document on the changes made on ASA NAT Exempt Rules for VPN tunnels between ASA version 8.2 and 8.3 (9.1 in my case). I tried to put whatever I could find on Cisco’s support site and on Google into my config prior to migration day, but of course what I had in there was wrong.

On ASA version 8.2 NAT exempt rules were simple; click the Add button under NAT Rules, followed by Add NAT Exempt Rule. Simple!

1

Enter your source interface, source host(s) and destination VPN network. Boom done!

2

If you were NAT’ing through the VPN tunnel you created a Static Policy NAT rule. Defined the source server, destination VPN network, and the translated address. A little more work, but still simple.

3

Then comes the new ASA version!! Ok I know ASA, let me go to my NAT Rules, Add… wait a minute…?

4

The new version doesn’t contain Policy NAT rules, NAT exempt rules or Dynamic NAT rules!! With a little Googling I couldn’t find any definitive answer as to how to create my NAT exempt rules for VPN. In both cases I exempt using the servers Real IP and I NAT to another IP over the tunnel.

Real IP NAT exempt

Here is how to do it through ASDM, I’ll post the command line results at the end.

  1. Click Add NAT Rule Before “Network Object” NAT Rules…
  2. 5
  3. Pick your inside interface where the server sits.
  4. Pick the interface (most likely outside) where the VPN traffic traverses through.
  5. Add a network object for Source Address; either subnet, range or host IP.
  6. Pick the destination IP/subnet that the VPN interested traffic would match.
  7. Check Disable Proxy ARP
  8. Leave the rest

6

A statement will show up above your network object NAT rules like this:

7

The command line equivalent would be this:

object network MY-RSERVER
 host 10.100.1.99
 object network VPN-NETWORK
 subnet 172.44.0.0 255.255.0.0
 nat (DEVPUB,OUTSIDE) 7 source static MY-RSERVER MY-RSERVER destination static VPN-NETWORK VPN-NETWORK no-proxy-arp

NAT Exempt with Translation

  1. Click Add NAT Rule Before “Network Object” NAT Rules…
  2. 5
  3. Pick your inside interface where the server sits.
  4. Pick the interface (most likely outside) where the VPN traffic traverses through.
  5. Add a network object for Source Address; either subnet, range or host IP.
  6. Pick the destination IP/subnet that the VPN interested traffic would match.
  7. To NAT the traffic, enter a network object with the translated IP in the Source Address field.
  8. Check Disable Proxy ARP

8

Once you enter the config, you will get a line in ASDM like this:

9

The resulting command line results will be the following.

object network MY-RSERVER
 host 10.100.1.99
object network MY-RSERVER_VPNNAT
 host 192.168.10.99
object network VPN-NETWORK
 subnet 172.44.0.0 255.255.0.0
nat (DEVPUB,OUTSIDE) 7 source static MY-RSERVER MY-RSERVER_VPNNAT destination static VPN-NETWORK VPN-NETWORK no-proxy-arp

Without these NAT statements your VPN traffic will not properly go over the tunnel.

Network object NAT Rule Changes

The second part that I was not used to was the way that NAT statements were configured for public servers. It was really confusing to me at first because there are so many options with the new version. Simply stated, this is how to do it.

  1. Go to the Network Objects pane
  2. 10
  3. Click Add
  4. Enter the name for your object and IP address
  5. Expand the NAT section and click Add Automatic Addresss Translation Rules
  6. Add a network object with the Public IP of the host11
  7. Expand the Advanced… section
  8. Check Disable Proxy ARP
  9. Pick the source interface the server sits on
  10. Pick the destination interface for your public NAT(probably OUTSIDE)12
  11. Click OK

Once the configuration is complete you will get something like this:

13

Command line equivalent:

object network MY-RSERVER
 host 10.100.1.99
object network MY-RSERVER_PUB
 host 4.2.2.1
object network MY-RSERVER
 nat (DEVPUB,OUTSIDE) static MY-RSERVER_PUB no-proxy-arp

OUTSIDE ACL Changes

The other issue that I ran into that was EXTREMELY frustrating was the ACL statements needed for the outside interface. In ASA version 8.2 it made sense: allow anyone, to my public IP, on the port I specify. I had all my ACL’s created on migration day, but nothing worked!! Why? Because on the new version your OUTSIDE ACL’s match against the real-IP of the server, not the NAT IP.

This is pretty self explanatory, once you figure it out. All you have to do is make sure your outside interface rules match against the REAL-IP of the server, not the public IP!

14

Command line equivalent:

access-list OUTSIDE_access_in line 1 extended permit tcp any object MY-RSERVER eq https
ASA Appliance to Module Headaches

Load balance based on CPU Load – (Windows 2008 Hosts)

Some load-balanced applications create considerable load independent to number of connections. For example reporting a server may become overloaded if users submit a report that requires a lot of historical data to generate. If there are three servers in a farm and they each have 10 connections, one server could have people running intensive reporting, while the other is sitting idle, which is imperceptible to the ACE with out some sort of CPU inspection. This is where SNMP probes come in…

The first thing we need to do is enable SNMP services. In my case I’m using Windows Server 2008 hosts. On Unix or other OS’s your process will vary or it may already be running SNMP services.

  • Open Server Manager
  • Click Features
  • Click Add Features on the right side
  • Scroll down and check the box marked SNMP Services
  • Click Next then Install

The next thing you need to do is configure it so that the ACE can talk to the host with the right strings.

  • Go back to Server Manager and expand Configuration
  • Click Services
  • Right click on SNMP Service and click Properties
  • Click the Security tab
  • Uncheck Send authentication trap
  • Under Accepted community names click Add...
  • Leave READ ONLY
  • Enter a community name appropriate for your environment (or use public) and click Add
  • Under Accept SNMP packets from these hosts click Add
  • Enter the IP of the ACE and click Add
  • Right click the service and click Restart

Now the ACE can poll the server for SNMP entries.

The first step is configuring the rserver(s) that are going to be monitored.

rserver host SERVER1
  ip address 192.168.1.1
  inservice
rserver host SERVER2
  ip address 192.168.1.2
  inservice

Now we want to build out our SNMP probe. The first step is to define it.

probe snmp CPU-PROBE

What was your community name? Enter it next.

community public

How often do you want the ACE to check the CPU? I used 10 seconds.

interval 10

If the server goes down, how many successful probes before it comes back online? Six would be 60 seconds, so I’ll use that.

passdetect count 6

The next section is tricky. How many CPU’s does your server have? You’ll have to customize your probe based on the number of CPU’s. In my case my server has two CPU’s. For one CPU the oid is .1.3.6.1.2.1.25.3.3.1.2.2 and the other is .1.3.6.1.2.1.25.3.3.1.2.3

What I’ll need to do is add both OID’s to the probe, then give them equal weight. In my example each CPU has a weight of 8000 (all your OID’s have to add up to 16000). If you had 8 CPU’s the weight would be 4000 each, and so on.

oid .1.3.6.1.2.1.25.3.3.1.2.2
 weight 8000
oid .1.3.6.1.2.1.25.3.3.1.2.3
 weight 8000

Now you can assign it to a serverfarm as a predictor method.

serverfarm MYFARM
  predictor least-loaded probe CPU-PROBE
  rserver SERVER1
    inservice
  rserver SERVER2
    inservice

The load is computed with the total weight of the probe, which is 16000. Run the show probe CPU-PROBE detail command to view the load on the server. Take that number and divide by 16000 to get the percentage value.

Here are some examples, the first one is for two-CPU servers and the second is for six-CPU servers.

Two CPU

probe snmp CPU-PROBE-TWO-CPU
 interval 10
 passdetect interval 60
 passdetect count 6
 community public
 oid .1.3.6.1.2.1.25.3.3.1.2.2
 weight 8000
 oid .1.3.6.1.2.1.25.3.3.1.2.3
 weight 8000

Six CPU

probe snmp CPU-PROBE-SIX-CPU
  interval 10
  passdetect interval 60
  passdetect count 6
  community public
  oid .1.3.6.1.2.1.25.3.3.1.2.2
    weight 2667
  oid .1.3.6.1.2.1.25.3.3.1.2.3
    weight 2666
  oid .1.3.6.1.2.1.25.3.3.1.2.4
    weight 2667
  oid .1.3.6.1.2.1.25.3.3.1.2.5
    weight 2667
  oid .1.3.6.1.2.1.25.3.3.1.2.6
    weight 2666
  oid .1.3.6.1.2.1.25.3.3.1.2.7
    weight 2667
Load balance based on CPU Load – (Windows 2008 Hosts)

HTTP to HTTPS Redirect on Cisco ACE

Instead of always having to write an HTTPS redirect on your webservers, you can have the ACE handle this work for you.

The first step is to create your redirection “rservers”. In my example it matches everything. If you want to only redirect a specific URL, then change the content after “https” to match what you want.

rserver redirect HTTPS-REDIRECT
  webhost-redirection https://%h%p 302
  inservice

Add to a redirect serverfarm:

serverfarm redirect FARM-HTTPS-REDIRECT
  rserver HTTPS-REDIRECT
     inservice

Once you have the serverfarm configured, apply to the respective policy-map. To match everything apply it to a class-default, if you want to redirect a specific URL assign it against a policy-map with multiple class definitions.

policy-map type loadbalance first-match POLICY-HTTPSREDIR
  class class-default
     serverfarm FARM-HTTPS-REDIRECT

Apply the policy map to your multi-match policy.

policy-map multi-match POLICY-STG1
 class CLASS-MYIP
   loadbalance vip inservice
   loadbalance policy POLICY-HTTPSREDIR
   loadbalance vip icmp-reply
 class SNAT
   nat dynamic 1 vlan 100

Add that to your interface and you’ll be good to go.

HTTP to HTTPS Redirect on Cisco ACE

ACE–why certificate names matter!!

I learned a lesson last night helping a co-worker with an ACE problem. The lesson was the importance of certificate names on ACE load balancers–specifically in an active/passive configuration.

Lets say you define your proxy service like this:

ssl-proxy service MyProxy
  key MyKey-2048
  cert My_Fun_cert.crt
  chaingroup MyChain
  ssl advanced-options MyCipher

In this configuration the certificate is called My_Fun_cert.crt. This exists on the ACE where it was defined and everything is happy. However, it may not on the passive ACE (our passive ACE the cert was a different name). This is easier to overlook when upgrading certs. If you upgrade and forget to upgrade the passive as well, big trouble is on the way…

In the scenario that I witnessed; the active ACE had its certificate upgraded, the passive did also, but it was named differently.

The primary ACE that was running happily for months decided to crash, causing a failover to the passive node. When this happened the certificate defined in the shared configuration did not exist in the passive ACE. Uh oh!! If you opened a browser connection to the VIP, specifically Firefox, you got the following error:

ssl_error_rx_record_too_long

Kind of cryptic, but it told me there was a problem with SSL proxying.

We realized that the certificate was in-fact updated, but it did not match what was on the primary ACE (sh crypto files, show crypto certificates file). Once that was straightened out:

crypto export WrongNameCert.crt
--Copy to notepad and paste to:
crypto import RightNameCert.crt
crypto delete WrongNameCert.crt

When the passive became the primary and things were not working, we moved again to the primary. Still no luck… After some time of analyzing the config, we realized that the proxy service definition was removed from all policy-maps!

So when the passive became the active without the right certificate names, the running-configuration was changed by the ACE.

The end-solution was to

  1. Fix the problem with the certificate names
  2. Add the correct certificate name to the proxy-service
  3. Re-add the proxy-service definitions to the policy-maps that need SSL encryption.
ACE–why certificate names matter!!