Flaw in Syn Attack Protection on non-updated Microsoft OSes, can lead to DoS

Summary

It is possible to mount a DoS attack against Windows 2000/2003 hosts where
the SYN attack protection has been enabled. The attacker can consume all
CPU resources of the victim host making it unresponsive.
While a standard SYN flood attack can make a single application server
unavailable, this attack can make the whole host unreachable.

Systems Affected

Windows 2003 without SP1
Windows 2000 SP4 without Update Roll-Up

Description

On Windows 2000/2003 the system administrator can enable a SYN Attack
protection mechanism on the TCP/IP by adding the value SynAttackProtect in
the registry key HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters.
If the value of SynAttackProtect is 2 the TCP/IP stack notifies a
listening socket only when the 3-way handshake has been completed and
tracks the ongoing 3-way handshakes by storing them in an hash table.
This way the backlog of the socket is defended from the SYN floods attacks.

SynAttackProtect is not enabled by default on the affected systems but has
been recommended by a number of articles:
http://support.microsoft.com/default.aspx?scid=kb;en-us;Q315669&sd=tech
http://www.microsoft.com/technet/security/topics/networksecurity/secdeny.mspx
http://msdn.microsoft.com/library/en-us/dnnetsec/html/HTHardTCP.asp
http://support.microsoft.com/default.aspx?scid=kb;en-us;142641
http://www.securityfocus.net/infocus/1729
http://www.awprofessional.com/articles/article.asp?p=371702

The vulnerability resides in the hash table management, in fact the hash
function used by the TCP/IP stack works only on some fields of the
incoming SYN packet and is thus predictable. An attacker can generate a
large number of SYN packets with the same hash value to target the same
hash table bucket. When the victim machine receives them, it stores them
in just one bucket of the hash table. The chain attached to this bucket
keeps growing, and the more it grows, the slower the lookup algorithm
becomes.

Vendor response

I've notified Microsoft of the vulnerability 2 years ago, when
the attack was possible on the Windows 2000 version (SP3) in production at
that time.
They confirmed the vulnerability but didn't release a patch because the
correction needed extensive changes in the code of the TCP/IP stack.
Microsoft has patched the vulnerability in Windows 2003 SP1 and
Windows 2000 Update Roll-up but it has inadvertently forgot to notify me.
The new version of TCPIP.SYS has this Syn Attack Protection enabled by
default but uses a crypto hash function (MD5) for the table lookup. The
hash material is the source port, dest port, source ip, dest ip of the SYN
packet and some pseudo random material extracted at startup.
This way the hash function is not easily predictable.


-- 
Luigi Mori

Symbolic S.p.A.
W: www.symbolic.it
T: +390521708811