From: "Lancashire, Andrew" <LancashireA@sutterhealth.org>

This is to clarify what is being put out by Cisco and what we are being told
by Cisco.

Two e-mails below is what Cisco is telling us and makes alot more sense
than what Cisco is telling Bugtraq. The last post to Bugtraq made mention
that the arp cache was filling up and allocating memory for both reachable
hosts and unreachable hosts (incompletes).  Although what Lisa describes is
true regarding the arp cache, it would not be true for our or most other
sane persons environment.  Since routers will only arp for what is local,
that would mean that for the arp cache to fill up and us all the memory all
networks in the 10.x.x.x range would need to be local.  So that's not gonna
happen but if you read the e-mail below that from Kenny (also at Cisco ) his
explanation makes allot more sense considering we have hundreds of routers.

Thank You 

Andrew

P.S. Congratulations on the re-opening of PacketStorm
___________________________________


Subject:      Re: Cisco and Nmap Dos
To: BUGTRAQ@SECURITYFOCUS.COM 


Hi all,


I wanted to address the items listed here.  We are still investigating this
problem, and hope to have more information later on in the week.  


Item 1, OSPF is not an issue.  According to the configuration information
provided to us by the customer, OSPF is not in use.


Items 2, 3 & 4.   IOS actually handles ARP in the following manner:  


When we receive a packet destined for something not already in our ARP
table, we  enter an "incomplete" entry in the ARP table.  Then we will rate
limit ARPs to once every 2 seconds to that destination.  Any additional
packets to that same destination will be dropped until the ARP entry is
completed.  This is on a per destination basis. 


"Incompletes" (ARP requests that have not been responded to) get dropped
after approximately 1 minute from the last time we sent an ARP request for
that non existent address.  


Incomplete entries MAY stay around longer, as the process that is
responsible for cleaning up the ARP table may not have enough time to
complete before it is called again, if we have a lot to clean up, which may
be relevant to this case.  The incomplete entries will eventually get
cleaned up, but it may take two or three minutes, two or three cycles of the
process that cleans up the table. 


Under a dedicated, intense nmap scan, a very large number of ARP requests
may be generated, causing the ARP table to grow very large with "incomplete"
entries.  These entries consume memory.  As the amount of free memory
declines and demand on the processor to handle outstanding packets
increases, ARP processing falls behind and throughput on the router may
decline significantly.  Once the scan is stopped, processing catches up and
things return to normal.


So, to my knowledge IOS should be doing the right thing, we only queue one
ARP request at a time, every 2 seconds, until the ARP entry is resolved, we
rate limit requests, dropping all additional packets, until the ARP entry is
resolved, and we clean up the outstanding incomplete requests within a few
minutes.  


I hope that helps address some of the concerns put forth.  Hopefully we will
have further updates shortly. 


Thank you,


Lisa Napier
Product Security Incident Response Team
Cisco Systems
___________

_______________

-----Original Message-----
From:  khollis [SMTP:khollis@cisco.com]
Sent:  Wednesday, September 15, 1999 11:31 AM
To:  wescotd@sutterhealth.org
Subject:  Regarding Case Number V44290

Hello Dave, I've done some testing here with Nmap. I was able to create a
test bed that can cause problems & symptoms similar to those you reported.
But in summary, the router is functioning normally & depending on the
network topology the behavior you experienced would be expected.

>From show processor memory, the ip input process is the process that
maintains the ip fast switching cache. This fast switching cache is used
when forwarding packets to avoid interrupting the cpu for repetitive route
table lookups. The key issue is the behavior of the fast cache and the way
it gets built.

There are a number of scenarios that will cause the fast cache to install an
entry that essentially looks like a host route. For instance, with only 1
path to a destination, we will install an entry into the fast cache that
covers the entire network. Example: 100.0.0.0/8. However, when multiple
equal cost paths to a destination exist, we will install an entry into the
fast cache for each destination. Example: 100.0.0.1/32, 100.1.1.1/32,
100.2.2.2/32...and so on. This helps ensure load balancing. Additionally,
depending on whether routes are directly connected, and/or subnetted, or the
next hop of a static route, the results can vary.

When running Nmap & scanning every address in a class A ip network, if
conditions warrant the installation of a /32 entry into the fast cache this
would allow the fast cache to consume a tremendous amount of memory and in
that scenario all available dram could be consumed. This creates additional
problems because there isn't enough memory to support other features on the
router.

Since Nmap isn't a normal application ran on networks, this issue isn't a
concern in most networking environments. However, if this is a major concern
you could run CEF (Cisco Express Forwarding). The behavior I just explained
does not occur when running CEF. But you will need to run 12.0 on the Cat5
RSM. Other workarounds such as disabling fast switching (no ip route-cache)
or using max-paths 1 aren't really feasible. CEF is the better solution.

Thanks,
KennyH.

_________________