Lately in the media has been some buzz [1] regarding the so called TCP split handshake, buzz triggered by the NSS Labs report Network Firewall 2011 Comparative Test Results.
The attack is described within the paper The TCP Split Handshake: Practical Effects on Modern Network Equipment [3].
It must be noted that some vendors quickly reacted to NSS Labs report, see Firewall Vendors Challenge Findings of NSS Labs Report [4].
WatchGuard’s explanation What is the TCP Split-Handshake Attack and Does It Affect Me [5] is pretty good if you want some extra details on the matter.
Fortinet has two blog entries about this subject: Fortinet Responds to NSS Labs Public Firewall Test [6] and On Tests, Firewalls and Modern Threat Mitigation [7].
Cisco also has a blog entry on this, Cisco Investigation for TCP Split-Handshake Issue Reported by NSS. Interesting this post received some extra coverage [9].
Check Point did not miss the opportunity and bragged about their product range [10].
On McAfee [11] and Juniper [12 forums questions were raised about the TCP split handshake issue. Other vendors may have received similar questions on their forums.
On the forums.isaserver.org also the very same question was posted [13] related to Forefront TMG 2010 .
Perhaps, apart the question if such a connection can go through TMG or be used by TMG itself, also of interest in case TCP split handshake would be allowed, is if the IPS(NIS), Malware Inspection or HTTP filtering can be affected by this.
Since there isn't an official answer from Microsoft yet, I decided to test myself this. My quick answer: no.
To detail a little bit, the underlying Windows OS, Forefront TMG 2010 was installed on Widows Server 2008 R2 SP1, completes with a crafty server such a handshake(tested with IE8 against a Backtrack 4 running the ruby script provided in [3]).
This is not necessarily a thing to worry about, since TMG is installed on top of the host OS, providing protection for this.
Quickly summarized, Forefront TMG 2010, by default, “blocks” such a response from the server passing through it, particularly blocks the lone SYN segment sent by the server, likely(can’t tell for sure as writing) considering it as a SYN segment used to start a new connection and normally there isn’t a rule to allow back such a new connection to the high port used by the client to initiate the connection(even so, TMG blocks such an attempt, see below); TMG uses a default deny all policy.
Note that Kernel mode forwarding (IP routing) is enabled by default on TMG [15]. According to [15] this means that:
When kernel mode forwarding is enabled, Forefront TMG creates separate connections between the client and server. Forefront TMG fully parses and then reconstructs the IP and TCP headers, transferring only the data parts. If a malicious client attempts to exploit an IP or TCP vulnerability, Forefront TMG blocks the traffic, and the traffic does not reach the destination computer, which is protected by Forefront TMG.
Since the ISA Server 2006 days, with IP routing enabled, according to [16]:
ISA Server validates that the three-way handshake packets required in a sequence are valid. This avoids establishing TCP connections to or through ISA Server from spoofed source IP addresses.
You need to understand that we cannot “just simply test/check” TMG against the TCP split handshake.
TMG’s behavior may vary per the mode of deployment of TMG and the level of inspection applied(stateful packet inspection, application filters inspection or IPS inspection to name some).
For example, deployed as an edge firewall with two NICs(WAN and LAN) doing NAT, regarding outbound TCP connections(LAN to Internet) a couple of things things happen.
On TMG there are some application filters, mostly such filters are not applied to access rules(used to allow “outbound access”), just on server publishing rules(used to publish an internal server located behind TMG), main exception being the web proxy, other ones being FTP or PPTP connections; the web proxy is special in that HTTP connections are always “NATed”(being proxied).
Also the NIS(IPS) has some protocol analyzers as well [14].
In this configuration, usually, TMG changes the source port used by the client behind it and also changes the SYN segment's sequence number; TMG maintains the TCP state of the connection and the lone SYN segment sent by the server as part of the 5-way or 4-way TCP Split-Handshake is dropped by TMG’s default rule. Also the lone ACK segment sent by the server is dropped(with Wireshark on the client we cannot see this segment reaching the client).
Note that in case of a HTTP request, even with the forward web proxy in transparent mode, the TCP connection between the client and server is established between the client and TMG(so with Wireshark on the client we will see the normal TCP three-way handshake), the web proxy does not complete the TCP handshake using the split handshake, the connection being denied, see the image from below.
Another example, say TMG deployed as a firewall with two NICs separating two networks, no NAT, routing relationship between the two networks on TMG. Regarding TCP connections passing through TMG things may vary from the NAT scenario.
If access rules are used to allow communication between the two networks, it depends if NIS protocol analyzers are used. For example, application layer traffic like POP3, SMTP, etc. can be inspected by NIS; in this case, TMG does not change anymore the source port but changes the SYN segment's sequence number and does not even allow the lone server ACK segment back to the original client, this is dropped along with the server’s lone SYN segment, see the first image from below. For other traffic that is not inspected by NIS, TMG does not change anymore the source port and the SYN segment's sequence number, allows the lone ACK from the server back to the client, but blocks the lone SYN. Again note that traffic served by the web proxy(like HTTP) will be proxied.
An odd situation may appear when the admin has allowed all traffic between the two networks in both directions. Even in this case TMG blocks the lone SYN with an error(maybe the port combo was already bind by TMG for the already initiated connection or so), see the second image from below.
On a final note, during my brief tests just the IPS was enabled on TMG, the malware solution was not enabled as requires a license(and the trial was expired on my test machine).
References
[1] Hacker 'handshake' hole found in common firewalls
http://www.networkworld.com/news/2011/041211-hacker-exploit-firewalls.html
[2] Network Firewall 2011 Comparative Test Results
https://www.nsslabs.com/research/network-security/firewall-ngfw/network-firewall-group-test-q2-2011.html
[3] The TCP Split Handshake: Practical Effects on Modern Network Equipment
http://nmap.org/misc/split-handshake.pdf
[4] Firewall Vendors Challenge Findings of NSS Labs Report
http://www.pcworld.com/businesscenter/article/225115/firewall_vendors_challenge_findings_of_nss_labs_report.html
[5] What is the TCP Split-Handshake Attack and Does It Affect Me?
http://watchguardsecuritycenter.com/2011/04/15/what-is-the-tcp-split-handshake-attack-and-does-it-affect-me/
[6] Fortinet Responds to NSS Labs Public Firewall Test
http://blog.fortinet.com/fortinet-responds-to-nss-labs-public-firewall-test/
[7] Cisco Investigation for TCP Split-Handshake Issue Reported by NSS
http://blogs.cisco.com/security/cisco-investigation-for-tcp-split-handshake-issue-reported-by-nss/
[8] On Tests, Firewalls and Modern Threat Mitigation
http://blog.fortinet.com/on-tests-firewalls-and-modern-threat-mitigation/
[9] Splits, handshakes, bananas and ransom notes
http://labs.mudynamics.com/2011/04/14/splits-handshakes-bananas-and-ransom-notes/
[10] Check Point First to Complete NSS Labs' Next Generation Firewall Test and Earn "Recommend" Rating
http://www.checkpoint.com/press/2011/NSS-NGFW-standalone-04.05.11.html
[11] McAfee forum: TCP Split Handshake Attack
https://community.mcafee.com/message/185055
[12] Juniper forum: TCP Split Handshake Attack
http://forums.juniper.net/t5/ScreenOS-Firewalls-NOT-SRX/TCP-Split-Handshake-Attack/td-p/85264
[13]forums. isaserver.org "TCP Split Handshake Attack" is TMG vulnerable?
http://forums.isaserver.org/%22TCP_Split_Handshake_Attack%22__is_TMG_vulnerable%3f/m_2002107715/tm.htm
[14] Forefront TMG 2010 NIS detection methods with signatures/protocol anomalies examples
http://www.carbonwind.net/Forefront_TMG/NIS_Det_Meth/NIS_Det_Meth.htm
[15] Overview of intrusion detection
http://technet.microsoft.com/en-us/library/cc995155.aspx
[16] ISA Server Network Protection: Protecting Against Floods and Attacks
http://technet.microsoft.com/en-us/library/bb794735.aspx