Debugging ARP on Cisco ASA

broadcast

The packet capture wizard in ASDM is a great feature of the ASA platform. It allows a network administrator to easily debug an issue and export the capture right to Wireshark from the wizard.

However, as you use this you may notice something. Where are my arp packets? Any time Wireshark is ran from a layer-2 network, arp packets will inevitably be captured. Something I didn’t know is that the ASDM wizard does not capture broadcast packets (at least at the time this was written ASA version 9.4(2) and ASDM 7.6).

Unfortunately Cisco doesn’t really describe this in any of their capture documentation, so if you don’t typically capture through the command line, you’ll never see broadcasts and may wonder what’s wrong.

How can I capture arp broadcasts on my ASA for troubleshooting layer-2 issues?

You have to do this through the ASA command line.

  1. Log in to the ASA you want to capture/see ARP packets.
  2. Use the ‘capture’ command with the ethernet-type arp

An example would be:

ASA# capture arp-cap ethernet-type arp interface inside

Where arp-cap is the name of your capture, the ethernet-type filters the capture to only arp packets and the interface picks the interface where you want to see the broadcasts.

You can define a ‘buffer’ flag if you want, but don’t worry about overloading your ASA, the default is 512kb. The above command is typically what you want.

Now we can execute a show command to see the capture buffer:

ASA# sh cap arp

81 packets captured

1: 13:21:17.283554 arp who-has 192.168.10.1 (cc:3:ca:f8:34:50) tell 192.168.10.21
 2: 13:21:17.283630 arp reply 192.168.10.1 is-at cc:3:ca:f8:34:50
 3: 13:21:18.600005 arp who-has 10.4.49.190 tell 192.168.10.1
 4: 13:21:20.053692 arp who-has 192.168.10.1 (cc:3:ca:f8:34:50) tell 192.168.10.167
 5: 13:21:20.053784 arp reply 192.168.10.1 is-at cc:3:ca:f8:34:50
 6: 13:21:21.069271 arp who-has 10.4.49.182 tell 10.4.48.25
 7: 13:21:21.998391 arp who-has 10.4.49.182 tell 10.4.48.25

We now  see who is broadcasting for what, and what hardware address they reside on. Use the detail flag to see more information.

Clean Up

We’ve got what we need, so its time to clean up. It’s very simple:

ASA# no cap arp-cap
Advertisements
Debugging ARP on Cisco ASA

ACL Debugging on ASA with ASDM

I know–that title is a mouthful, but it’s actually pretty simple. A lot of environments out there have ACL’s that have implicit permits because it was simply too much work to get a list of what needs to talk to what before the firewall(s) went into place.

So, you’ve got an ACL applied to an interface that is set to permit all. The goal is to not permit everything and break out individual permit rules so we can change that to a deny rule. In ASDM it should look something like this.

1

So what goes through that ACL that I can start breaking out and writing specific rules for?

Right now if you right click the ACL in ASDM you get this nice little function called “Show Log…” The problem is this doesn’t work right out of the gate. If you click on this option you wont see anything yet.

2

The first thing I like to do is enable Debug logging on ASDM. That way when I open the log viewer or try to see hits on an ACL–I don’t have to modify anything.

Under Configuration –> Logging –> Logging Filters –> ASDM change “Filter on severity” to Debugging

The strange thing is now when you go back to the ACL’s and click “Show Log…” you still don’t see anything. Why? The problem is that permit statements will not log by default. This needs to be enabled either in ASDM or through the command line.

Command Line:

logging message 106100

ASDM:

Configuration > Device Management > Logging > Syslog Setup > 106100 “Disabled = No”

Now when you right click a rule and select “Show Log…” you will start seeing hits on the ACL. Monitor these hits for a while during work hours and you will at least get a huge chunk of the rules written. This is really nice for breaking out permit rules so you can eventually have an implicit Deny All there instead of a Permit All.

ACL Debugging on ASA with ASDM

ASA Appliance to Module Headaches

asa-smheadache

I recently migrated one of our environments from an ASA 55xx appliance to an ASA service module running in a 6509-e chassis. There were a few gotchas that I was not aware of that made the transition date quite an interesting experience… There were three pain-points for me on migration day. I’ll cover what they were and the configuration that was actually needed to make things work.

I use ASDM for most of my day-to-day work on ASA, so I’ll post the configuration with ASDM with the command line equivalent after.

NAT Exempt rules for VPN

I had a heck of a time finding a definitive document on the changes made on ASA NAT Exempt Rules for VPN tunnels between ASA version 8.2 and 8.3 (9.1 in my case). I tried to put whatever I could find on Cisco’s support site and on Google into my config prior to migration day, but of course what I had in there was wrong.

On ASA version 8.2 NAT exempt rules were simple; click the Add button under NAT Rules, followed by Add NAT Exempt Rule. Simple!

1

Enter your source interface, source host(s) and destination VPN network. Boom done!

2

If you were NAT’ing through the VPN tunnel you created a Static Policy NAT rule. Defined the source server, destination VPN network, and the translated address. A little more work, but still simple.

3

Then comes the new ASA version!! Ok I know ASA, let me go to my NAT Rules, Add… wait a minute…?

4

The new version doesn’t contain Policy NAT rules, NAT exempt rules or Dynamic NAT rules!! With a little Googling I couldn’t find any definitive answer as to how to create my NAT exempt rules for VPN. In both cases I exempt using the servers Real IP and I NAT to another IP over the tunnel.

Real IP NAT exempt

Here is how to do it through ASDM, I’ll post the command line results at the end.

  1. Click Add NAT Rule Before “Network Object” NAT Rules…
  2. 5
  3. Pick your inside interface where the server sits.
  4. Pick the interface (most likely outside) where the VPN traffic traverses through.
  5. Add a network object for Source Address; either subnet, range or host IP.
  6. Pick the destination IP/subnet that the VPN interested traffic would match.
  7. Check Disable Proxy ARP
  8. Leave the rest

6

A statement will show up above your network object NAT rules like this:

7

The command line equivalent would be this:

object network MY-RSERVER
 host 10.100.1.99
 object network VPN-NETWORK
 subnet 172.44.0.0 255.255.0.0
 nat (DEVPUB,OUTSIDE) 7 source static MY-RSERVER MY-RSERVER destination static VPN-NETWORK VPN-NETWORK no-proxy-arp

NAT Exempt with Translation

  1. Click Add NAT Rule Before “Network Object” NAT Rules…
  2. 5
  3. Pick your inside interface where the server sits.
  4. Pick the interface (most likely outside) where the VPN traffic traverses through.
  5. Add a network object for Source Address; either subnet, range or host IP.
  6. Pick the destination IP/subnet that the VPN interested traffic would match.
  7. To NAT the traffic, enter a network object with the translated IP in the Source Address field.
  8. Check Disable Proxy ARP

8

Once you enter the config, you will get a line in ASDM like this:

9

The resulting command line results will be the following.

object network MY-RSERVER
 host 10.100.1.99
object network MY-RSERVER_VPNNAT
 host 192.168.10.99
object network VPN-NETWORK
 subnet 172.44.0.0 255.255.0.0
nat (DEVPUB,OUTSIDE) 7 source static MY-RSERVER MY-RSERVER_VPNNAT destination static VPN-NETWORK VPN-NETWORK no-proxy-arp

Without these NAT statements your VPN traffic will not properly go over the tunnel.

Network object NAT Rule Changes

The second part that I was not used to was the way that NAT statements were configured for public servers. It was really confusing to me at first because there are so many options with the new version. Simply stated, this is how to do it.

  1. Go to the Network Objects pane
  2. 10
  3. Click Add
  4. Enter the name for your object and IP address
  5. Expand the NAT section and click Add Automatic Addresss Translation Rules
  6. Add a network object with the Public IP of the host11
  7. Expand the Advanced… section
  8. Check Disable Proxy ARP
  9. Pick the source interface the server sits on
  10. Pick the destination interface for your public NAT(probably OUTSIDE)12
  11. Click OK

Once the configuration is complete you will get something like this:

13

Command line equivalent:

object network MY-RSERVER
 host 10.100.1.99
object network MY-RSERVER_PUB
 host 4.2.2.1
object network MY-RSERVER
 nat (DEVPUB,OUTSIDE) static MY-RSERVER_PUB no-proxy-arp

OUTSIDE ACL Changes

The other issue that I ran into that was EXTREMELY frustrating was the ACL statements needed for the outside interface. In ASA version 8.2 it made sense: allow anyone, to my public IP, on the port I specify. I had all my ACL’s created on migration day, but nothing worked!! Why? Because on the new version your OUTSIDE ACL’s match against the real-IP of the server, not the NAT IP.

This is pretty self explanatory, once you figure it out. All you have to do is make sure your outside interface rules match against the REAL-IP of the server, not the public IP!

14

Command line equivalent:

access-list OUTSIDE_access_in line 1 extended permit tcp any object MY-RSERVER eq https
ASA Appliance to Module Headaches

Adding Routes on Windows Servers

I’ll preface this post by saying I HATE host routes (not often that I capitalize words so you know how serious this is…) Lets face it, sometimes you need to add one to make an old and a new environment work, testing, VPN, B2B networks, etc. etc.

Windows has a basic, but in my opinion, a somewhat lousy command tool called “route” to add host routes. In it’s most basic form and under most circumstances you can use it right out of the box without any issues.

There are, however, those times when a server has three NIC’s, you need to add routes for different subnet masks, specific hosts, etc. That’s where things can get tricky.

I recently encountered an issue where even though we added a host route, it did not work. I knew something was wrong because when I captured traffic on the firewall that was the gateway for the route, I just didn’t see anything. Here’s the scenario; I am on the 10.100.1.0 subnet with 10.100.1.1 as my default server gateway. I need to talk to 10.100.2.50 but through a different gateway.

The first thing I did was add a one-to-one host route:

route add -p 10.100.2.50 mask 255.255.255.255 10.100.1.2

Ok, so that’s it right? Nope. By default it assigns interface 1 to the route, which is the loopback interface. When I tried to ping that 10.100.2.50 host I never saw the traffic.

I had to run a route print to see what my interfaces were numbered as:

C:\Users\me>route print
=============================================================
Interface List
 17...00 50 46 84 00 13 ......vmxnet3 Ethernet Adapter #2
 14...00 0c c9 0d d2 da ......vmxnet3 Ethernet Adapter #1
 1...........................Software Loopback Interface 1
=============================================================

Ok, so ethernet Adapter 1 is the one I want to use, which is interface 14, so I need to adjust my route statement.

route add -p 10.100.2.50 mask 255.255.255.255 10.100.1.2 if 14

Nice! My traffic works now.

Why do I think the route command is lousy? Well, because when I add a route for a /24 subnet, it works 90% of the time, but when I add a specific host (/32) route, I have to specify the interface. For example:

route add -p 10.100.2.0 mask 255.255.255.0 10.100.1.2

This command normally would work without specifying any interfaces. Why? I have no idea, maybe some Microsoft employees can fill me in.

The other issue I have is the inability to ping using a different gateway. I can ping using a different IP on that host with the ping -S command, but there is no way for me to test a ping without messing about with routes until I see what I need on my firewall. I would love it if I could ping -G and use a gateway IP to send traffic. Alas, I am getting off topic…

Adding Routes on Windows Servers

Load balance based on CPU Load – (Windows 2008 Hosts)

Some load-balanced applications create considerable load independent to number of connections. For example reporting a server may become overloaded if users submit a report that requires a lot of historical data to generate. If there are three servers in a farm and they each have 10 connections, one server could have people running intensive reporting, while the other is sitting idle, which is imperceptible to the ACE with out some sort of CPU inspection. This is where SNMP probes come in…

The first thing we need to do is enable SNMP services. In my case I’m using Windows Server 2008 hosts. On Unix or other OS’s your process will vary or it may already be running SNMP services.

  • Open Server Manager
  • Click Features
  • Click Add Features on the right side
  • Scroll down and check the box marked SNMP Services
  • Click Next then Install

The next thing you need to do is configure it so that the ACE can talk to the host with the right strings.

  • Go back to Server Manager and expand Configuration
  • Click Services
  • Right click on SNMP Service and click Properties
  • Click the Security tab
  • Uncheck Send authentication trap
  • Under Accepted community names click Add...
  • Leave READ ONLY
  • Enter a community name appropriate for your environment (or use public) and click Add
  • Under Accept SNMP packets from these hosts click Add
  • Enter the IP of the ACE and click Add
  • Right click the service and click Restart

Now the ACE can poll the server for SNMP entries.

The first step is configuring the rserver(s) that are going to be monitored.

rserver host SERVER1
  ip address 192.168.1.1
  inservice
rserver host SERVER2
  ip address 192.168.1.2
  inservice

Now we want to build out our SNMP probe. The first step is to define it.

probe snmp CPU-PROBE

What was your community name? Enter it next.

community public

How often do you want the ACE to check the CPU? I used 10 seconds.

interval 10

If the server goes down, how many successful probes before it comes back online? Six would be 60 seconds, so I’ll use that.

passdetect count 6

The next section is tricky. How many CPU’s does your server have? You’ll have to customize your probe based on the number of CPU’s. In my case my server has two CPU’s. For one CPU the oid is .1.3.6.1.2.1.25.3.3.1.2.2 and the other is .1.3.6.1.2.1.25.3.3.1.2.3

What I’ll need to do is add both OID’s to the probe, then give them equal weight. In my example each CPU has a weight of 8000 (all your OID’s have to add up to 16000). If you had 8 CPU’s the weight would be 4000 each, and so on.

oid .1.3.6.1.2.1.25.3.3.1.2.2
 weight 8000
oid .1.3.6.1.2.1.25.3.3.1.2.3
 weight 8000

Now you can assign it to a serverfarm as a predictor method.

serverfarm MYFARM
  predictor least-loaded probe CPU-PROBE
  rserver SERVER1
    inservice
  rserver SERVER2
    inservice

The load is computed with the total weight of the probe, which is 16000. Run the show probe CPU-PROBE detail command to view the load on the server. Take that number and divide by 16000 to get the percentage value.

Here are some examples, the first one is for two-CPU servers and the second is for six-CPU servers.

Two CPU

probe snmp CPU-PROBE-TWO-CPU
 interval 10
 passdetect interval 60
 passdetect count 6
 community public
 oid .1.3.6.1.2.1.25.3.3.1.2.2
 weight 8000
 oid .1.3.6.1.2.1.25.3.3.1.2.3
 weight 8000

Six CPU

probe snmp CPU-PROBE-SIX-CPU
  interval 10
  passdetect interval 60
  passdetect count 6
  community public
  oid .1.3.6.1.2.1.25.3.3.1.2.2
    weight 2667
  oid .1.3.6.1.2.1.25.3.3.1.2.3
    weight 2666
  oid .1.3.6.1.2.1.25.3.3.1.2.4
    weight 2667
  oid .1.3.6.1.2.1.25.3.3.1.2.5
    weight 2667
  oid .1.3.6.1.2.1.25.3.3.1.2.6
    weight 2666
  oid .1.3.6.1.2.1.25.3.3.1.2.7
    weight 2667
Load balance based on CPU Load – (Windows 2008 Hosts)

VPN Tunnel Visio Diagram – Template

I do the same thing with VPN tunnels as I do with VLAN diagrams. Managers, developers, sys. admins always ask about port configurations, destinations, contact information, etc. It’s a nice guide general enough to understand but detailed enough to understand as a non-techie.

If you want a VLAN diagram to supplement this, I have a template here.

Diagram here

VPN Tunnel Template

VPN Tunnel Visio Diagram – Template