Category Archives: Networking

Cisco WAP571 SNMP poll of apRadioNumAssociatedStations returning 0

For some reason on a Cisco WAP571, the SNMP value returned from apRadioNumAssociatedStations is always zero.

This is true on firmware tested WAP571 (pgwap571, 1.1.0.3).

I have a few of these units around that are not updated to the latest firmware and will test that OID.

I can find data in the apAssocTable to create indexes. For now I’ve created a hack to just snmptable the apAssocTable and count return index values to then pass to Cacti to graph.

snmptable -Cl -CB -Ci -OX -Cb -Cc 16 -Cw 64 -v2c -c <community> <host:port> CISCO-WLAN-ACCESS-POINT-MIB::apAssocTable | grep index | wc -l

snmptable

snmptable -Cl -CB -Ci -OX -Cb -Cc 16 -Cw 64 -v2c -c <community> <host:port> CISCO-WLAN-ACCESS-POINT-MIB::apAssocTable

On a Cisco WAP571, it produces the following output.

Interface       Authenticated   Associated      RxPackets
TxPackets       RxBytes         TxBytes         ListenInterval
LastRssi        TxDropBytes     RxDropBytes     TxDropPackets
RxDropPackets   ClntQoSStatus   BwLimitUp       BwLimitDown
ACLType         ACL             Policy          TsViolateTxPack
TsViolateRxPack Uptime

index: [8:c5:e1:35:b5:89]
wlan1vap2       yes             yes             17056
23016           3871353         25017797        10
32              0               0               0
0               Disabled        0               0
None                                            Wrong Type (sho
Wrong Type (sho 00:01:03

index: [48:90:2f:cd:9f:35]
wlan1vap2       yes             yes             75219
433258          14643877        530054579       2
22              0               0               0
0               Disabled        0               0
None                                            Wrong Type (sho
Wrong Type (sho 00:02:10

index: [7c:f3:1b:bf:ca:ff]
wlan0vap2       yes             yes             14351
23055           1897849         28821887        5
15              0               0               0
0               Disabled        0               0
None                                            Wrong Type (sho
Wrong Type (sho 00:00:36

index: [8c:83:e1:ce:33:f2]
wlan1vap2       yes             yes             91
93              15906           37253           5
39              0               0               0
0               Disabled        0               0
None                                            Wrong Type (sho
Wrong Type (sho 00:00:19

index: [b8:d7:af:f:47:4c]
wlan1vap2       yes             yes             9948
8279            1935919         3804742         10
30              0               0               0
0               Disabled        0               0
None                                            Wrong Type (sho
Wrong Type (sho 00:06:30

ISAKMP (IKE Phase 1) Status Messages MM_WAIT_MSG#

I’ve been meaning to copy this back here but haven’t had the chance until now. I reference this so much, figured it’d have stuck in my mind by now… Anyway, this is one of the best resources for quick analysis troubleshooting of MM_WAIT_MSG errors on VPN tunnels for Cisco ASA / PIX from https://www.tunnelsup.com/isakmp-ike-phase-1-status-messages/.

ISAKMP (IKE Phase 1) Negotiations States

The MM_WAIT_MSG state can be an excellent clue into why a tunnel is not forming. If your firewall is hanging at a specific state review this graph below to find where along the path the VPN is failing.

ASA ISAKMP STATES

Graph source: tunnelsup.com


These are the possible ISAKMP negotiation states on an ASA firewall. ISAKMP stands for: The Internet Security Association and Key Management Protocol

  • MM_WAIT_MSG2 Initiator Initial DH public key sent to responder. Awaiting initial contact reply from other side. Initiator sends encr/hash/dh ike policy details to create initial contact. Initiator will wait at MM_WAIT_MSG2 until it hears back from its peer. If stuck here it usually means the other end is not responding. This could be due to no route to the far end or the far end does not have ISAKMP enabled on the outside or the far end is down.
  • MM_WAIT_MSG3 Receiver Receiver is sending back its IKE policy to the initiator. Initiator sends encr/hash/dh ike policy details to create initial contact. Initiator will wait at MM_WAIT_MSG2 until it hears back from its peer. Hang ups here may also be due to mismatch device vendors, a router with a firewall in the way, or even ASA version mismatches.
  • MM_WAIT_MSG4 Initiator Initiator is sending the Pre-Shared-Key hash to its peer. Initiator sends a hash of its PSK. Initiator will stay at MSG4 until it gets a PSK back from its peer. If the receiver is missing a tunnel group or PSK the initiator will stay at MM_WAIT_MSG4
  • MM_WAIT_MSG5 Receiver Receiver is sending its PSK hash to its peer. Receiver does not yet check if PSK hashes match. If receiver has a tunnel-group and PSK configured for this peer it will send the PSK hash to the peer. If PSKs don’t match, receiver will stay at MM_WAIT_MSG5. I have also seen the tunnel stop here when NAT-T was on when it needed to be turned off.
  • MM_WAIT_MSG6 Initiator Initiator checks if PSK hashes match. If PSK keys match, Initiator becomes MM_ACTIVE and lets receiver know of match. If PSK doesn’t match, initiator stays at MM_WAIT_MSG6. I have also seen the tunnel stop here when NAT-T was on when it needed to be turned off. However, if the state goes to MSG6 then the ISAKMP gets reset that means phase 1 finished but phase 2 failed. Check that IPSEC settings match in phase 2 to get the tunnel to stay at MM_ACTIVE.
  • AM_ACTIVE / MM_ACTIVE The ISAKMP negotiations are complete. Phase 1 has successfully completed.

PIX ISAKMP STATES

  • MM_NO_STATE

ISAKMP SA has been created but nothing else has happened yet.

  • MM_SA_SETUP

The peers have agreed on parameters for the ISAKMP SA.

  • MM_KEY_EXCH

The peers have exchanged Diffie-Hellman public keys and have generated a shared secret. The I SAKMP SA remains unauthenticated.

  • MM_KEY_AUTH

The ISAKMP SA has been authenticated. If the router initiated this exchange, this state trans itions immediately to QM_IDLE and a Quick mode exchange begins.

  • AG_NO_STATE

The ISAKMP SA has been created but nothing else has happened yet.

  • AG_INIT_EXCH

The peers have done the first exchange in Aggressive mode but the SA is not authenticated.

  • AG_AUTH

The ISAKMP SA has been authenticated. If the router initiated this exchange, this state transitions immediately to QM_IDLE and a Quick mode exchange begins.

  • QM_IDLE

The ISAKMP negotiations are complete. Phase 1 successfully completed. It remains authenticated with its peer and may be used for subsequent Quick mode exchanges.

What is the difference between MM and AM?

Main mode vs Aggressive mode. Here is a image taken from Cisco’s website to show the difference.

Graphic source: tunnelsup.com


As you can see the Main mode is the same as the flowchart at the top of the page. Aggressive mode only uses 4 steps to establish the tunnel.

Troubleshooting ISAKMP Or Phase 1 VPN connections

When troubleshooting VPNs, a very common problem is phase 1 not establishing correctly. Here’s a quick checksheet to make sure you have the configuration correct.

  • Verify ISAKMP parameters match exactly.
  • Verify pre-shared-keys match exactly.
  • Check that each side has a route to the peer address that you are trying to form a tunnel with.
  • Verify ISAKMP is enabled on the outside interfaces.
  • Is ESP traffic permitted in through the outside interface?
  • Is UDP port 500 open on the outside ACL?
  • Some situations require that UDP port 4500 is open for the outside.

Warning, DHCP pool range is limited to 128 addresses

For all ASA models, the maximum number of DHCP client addresses varies depending on the license:

  • If the limit is 10 hosts, the maximum available DHCP pool is 32 addresses.
  • If the limit is 50 hosts, the maximum available DHCP pool is 128 addresses.
  • If the number of hosts is unlimited, the maximum available DHCP pool is 256 addresses.

That’s annoying.

The vpn client agent was unable to create the interprocess communication depot.

When installing Cisco AnyConnect VPN client, encountered an error:

The vpn client agent was unable to create the interprocess communication depot.

This error is due to Internet Connection Sharing being enabled.  To resolve, disable ICS per adapter, or globally through Services.
Per Adapter:

  1. Click the Start button.
  2. Click on Control Panel.
  3. Click on View Network Status and Tasks
  4. Click on Change adapter settings
  5. Right-click the shared connection and choose Properties
  6. Click the Sharing tab
  7. Clear the Allow other network users to connect through this computer’s Internet connection checkbox
  8. Click OK

System Wide:

  1. Click the Start button (Windows’ orb)
  2. Type: services.msc and press ENTER
  3. Double-Click on Internet Connection Sharing (ICS)
  4. Change Startup Type to Disabled
  5. Reboot the computer

 

Enable SSH Login on a Cisco Router

Quick example of setting up SSH access on a Cisco router. I have a few dozen routers in my lab I’m working on and actually made this scripted. This is here for me to remember in the future.

Router(config)# crypto key generate rsa usage-keys label rtr-key
The name for the keys will be: rtr-key
Choose the size of the key modulus in the range of 360 to 2048 for your
Signature Keys. Choosing a key modulus greater than 512 may take
a few minutes.
How many bits in the modulus [512]: 1024
Choose the size of the key modulus in the range of 360 to 2048 for your
Encryption Keys. Choosing a key modulus greater than 512 may take
a few minutes.
% Generating 1024 bit RSA keys, keys will be non-exportable...[OK]
Router(config)#exit

Check to make sure SSH is now enabled.

Router(config)# do sh ip ssh
SSH Enabled - version 2.0
Authentication timeout: 120 secs; Authentication retries: 3

Configure access now, setting SSH to perferred transport.

Router#conf t
!
line vty 0 4
access-class 1 in
exec-timeout 30 0
privilege level 15
login local
transport preferred ssh
transport input ssh
!

Go play.

Configure Cisco ASA to Capture Specific Port Traffic

On a Cisco ASA you can configure capturing of data to allow for deeper troubleshooting of issues. With the recent issue of the Heartbleed bug, I needed a way to capture HTTPS traffic and inspect remote hosts for the vulnerability. If the site was vulnerable, I would create a temporary block until that site patched.
On the Cisco ASA I setup an access-list:

access-list heartbleed line 1 extended permit tcp any any eq https

I create a capture:

capture heartbleed access-list heartbleed interface inside

Then I can view the capture:

show capture heartbleed

Example output of the above command:

1025: 09:52:27.882385 10.147.204.104.55665 > 74.125.228.5.443: . ack 3734113485 win 64860
1026: 09:52:27.882858 192.168.1.104.55666 > 74.125.228.5.443: . ack 3798098736 win 64860
1027: 09:52:27.883239 192.168.1.104.55666 > 74.125.228.5.443: . ack 3798101496 win 64860
1028: 09:52:27.883438 192.168.1.104.55666 > 74.125.228.5.443: . ack 3798104256 win 64860

Alternatively, while the capture is enabled it is accessible via the web interface of the ASA.

https://192.168.1.1/admin/capture/heartbleed

Test Cisco ASA VPN Authentication

Had an issue with a user that was failing to log into the VPN from remote.  Couldn’t initially figure it out while at home while troubleshooting the authentication.  So here’s how to test authentication from the Cisco ASA CLI.

ciscoasa# test aaa-server authentication AUTH2K8 host 192.168.1.2 username rkreider password s3cr3t

The blue highlights are values that need specified. If not sure of the AAA-SERVER, use the following command to list all the authentication servers.

ciscoasa# show aaa-server

This lists all the aaa-servers; to narrow it down, as in my case, I specified some additional arguments.

ciscoasa# show aaa-server authentication protocol nt

Here is a list of available protocols.

ciscoasa# show aaa-server protocol ?
  http-form  Protocol HTTP form-based
  kerberos   Protocol Kerberos
  ldap       Protocol LDAP
  nt         Protocol NT
  radius     Protocol RADIUS
  sdi        Protocol SDI
  tacacs+    Protocol TACACS+

So the output from showing the aaa-server type of NT is follows for me.

Server Group:    AUTH2K8
Server Protocol: nt
Server Address:  192.168.1.2
Server port:     139
Server status:   ACTIVE, Last transaction at 13:16:58 EDT Wed Mar 26 2014
Number of pending requests              0
Average round trip time                 0ms
Number of authentication requests       435
Number of authorization requests        0
Number of accounting requests           0
Number of retransmissions               0
Number of accepts                       389
Number of rejects                       31
Number of challenges                    0
Number of malformed responses           0
Number of bad authenticators            0
Number of timeouts                      15
Number of unrecognized responses        0

I used the highlighted values in my test case. Again, here is my command.

ciscoasa# test aaa-server authentication AUTH2K8 host 192.168.1.2 username rkreider password s3cr3t
INFO: Attempting Authentication test to IP address <192.168.1.2> (timeout: 12 seconds)
ERROR: Authentication Rejected: AAA failure
ciscoasa# test aaa-server authentication AUTH2K8 host 192.168.1.2 username rkreider password sup3rs3cr3t
INFO: Attempting Authentication test to IP address <192.168.1.2> (timeout: 12 seconds)
INFO: Authentication Successful

My issue was actually related to a setting on the account profile in Active Directory restricting server logons which inherently prevented authentication from working.

Cisco IOS EEM: Send Email on VPN Connection

I set up a Cisco router to send an email whenever a VPN user connected.  I did this for accounting purposes before I moved to RADIUS.  I’ll put this up here because someone else may be interested in this for their own use.

Step 1:  Environment Variable Setup

I like to configure variables to use throughout my EEM applets so I don’t get crazy with having to remember everything.  These setup a few such as a mail server, from email, to email.

router(config)#event manager environment _email_server 192.168.1.10
router(config)#event manager environment _email_from alerts@domain.local
router(config)#event manager environment _email_to admin@domain.local

Step 2: Create Event Manager Applet

Creating the applet is quite simple.

router(config)#event manager applet audit-vpn-login-ok

This creates the applet and puts you into its config mode to allow you to configure additional information.

Step 3: Identify what to look for

I simply look for a syslog pattern that corresponds to a Virtual-Access adapter being created (which indicates in my setup that a VPN has been established successfully).

router(config-applet)#event syslog pattern "LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access.*up"

After I tell it what to look for, I give it some actions.

Step 4: Configure Actions

For information purposes, I run a “show crypto ipsec sa | include local crypto” which stores the output of that command to a $_cli_results variable. This is helpful for telling me which IP address created the VPN. I could further this by looking for a username segment, but in this simple example, I’m sticking to just what IP established the VPN.

router(config-applet)#action 1.0 cli command "enable"
router(config-applet)#action 1.5 cli command "sh crypto ipsec sa | i local crypto"

Now I send the Email using the variables defined above and also include the $_cli_result (output of the command above stored as a variable) in the body.

router(config-applet)#action 2.0 mail server "$_email_server" to "$_email_to" from "$_email_from" subject "$_event_pub_time: VPN User Connected" body "Connection:n$_cli_result"

Finally, I send a syslog message notifying a VPN connection established as well.

router(config-applet)#action 2.5 syslog priority notifications msg "VPN UP - Mail Sent"

Full Code

router(config)#event manager environment _email_server 192.168.1.10
router(config)#event manager environment _email_from alerts@domain.local
router(config)#event manager environment _email_to admin@domain.local
router(config)#event manager applet audit-vpn-login-ok
router(config-applet)#event syslog pattern "LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access.*up"
router(config-applet)#action 1.0 cli command "enable"
router(config-applet)#action 1.5 cli command "sh crypto ipsec sa | i local crypto"
router(config-applet)#action 2.0 mail server "$_email_server" to "$_email_to" from "$_email_from" subject "$_event_pub_time: VPN User Connected" body "Connection:n$_cli_result"
router(config-applet)#action 2.5 syslog priority notifications msg "VPN UP - Mail Sent"