14.02.2011
Forefront TMG 2010 NIS detection methods with signatures/protocol anomalies examples

 - 1. A little bit of history
 - 1.1 IDS vs IPS
 - 1.2 IDS/IPS detection methods
 - 1.2.1 Signature-based detection
 - 1.2.1.1 Exploit-based signatures
 - 1.2.1.2 Vulnerability-based signatures
 - 1.2.1.3 Vulnerability-based signatures vs 0-day vulnerabilities
 - 1.2.1.4 Exploit-based signatures vs vulnerability-based signatures
 - 1.2.2 Anomaly-based detection
 - 1.2.3 Stateful protocol analysis detection
 - 1.2.4 Virtual patching using IPS
 - 1.3 Time frames(or exposure windows) related to a vulnerability and IPS
 - 2. Where TMG's NIS stands
 - 2.1 NIS signature examples
 - 2.1.1 A look at a NIS exploit-based signature
 - 2.1.1.1 Attempt to bypass the exploit-based signature(alternative publicly available PoC 1)
 - 2.1.1.1 Attempt to bypass the exploit-based signature(alternative publicly available PoC 2)
 - 2.1.2 A look at a NIS vulnerability-based signature
 - 2.1.2.1 Attempts to bypass the vulnerability-based signature
 - 2.1.2.1.1 Using URL encoding for special characters
 - 2.1.2.1.2 Using hex encoding for a random character
 - 2.1.2.1.3 Using double-byte encoding for a random character
 - 2.1.2.1.4 Using triple-byte encoding for a random character
 - 2.1.2.2 A few thoughts
 - 2.3 A look at a NIS policy-based signature
 - 2.4 Protocol anomaly detection examples
 - 3. Conclusion

 1. A little bit of history
Before we proceed and discuss about TMG's NIS, let's take a simplistic look back at the evolution of IDS and IPS solutions.

 1.1 IDS vs IPS
Today the words IPS and IDS are often interchanged.

Earlier basic IDS(Intrusion Detection Systems) solutions were devices sitting in passive mode(promiscuous mode) working on copies of data traffic.
They monitored and analyzed traffic looking for suspicious activities such as port scanning and being able to detect attacks based on signatures of known exploits.
Because they worked on copies of traffic they did not affect the traffic flow.
Originally an IDS detected attacks, raised alerts and recorded information related to the events observed.
IDS devices were deployed to address firewall limitations in terms of inspection capabilities, over the time being able to detect various types of attacks, like network attacks, attacks on applications, host-based attacks(brute force password guessing, unauthorized logins(root login over telnet)), Trojan horses, worms, etc.

Obviously it would have been useful to block attacks and not just raise alerts. So the IDS started to morph into an IPS.

However an IDS being a passive device, an active response action(block the attacker(per IP address), block the connection or reset the TCP connection etc.) was implemented with the help of networking devices(firewalls or routers). For example a TCP connection between the attacker and victim could have been terminated by a firewall upon receiving an alert message from the IDS.
To be noted that such a response action can only prevent some classes of attacks because the response action is a post-event response and thus lets malicious packets reach the victim. Imagine a single packet attack, the malicious packet would reach pretty much in the same time the victim(original packet) and the IDS(copy of that packet), by the time the IDS informs the firewall to take some actions, the malicious packet already arrived at its destination(victim).
An example of such a solution was SnortSam, a plugin for Snort(the popular open-source light-weight IDS, plugin which allowed for automated blocking of IP addresses on certain firewalls).
SnortSam could not block single packet attacks and had problems with "few packets attacks".

To address these limitations, IPS solutions were deployed.
Unlike an IDS, the IPS sits in inline mode, directly into the packet flow, working on forwarded traffic. An IPS would not forward packets unless they are "clean", thus theoretically no malicious traffic is allowed to pass(note that although the IPS does packet buffering, some packets(say part of a message), based on some assumptions may be actually forwarded(without the IPS have the complete message) to the protected host in the process).
The IPS can respond immediately, thus can block single packet attacks.
Because it sits inline the IPS has to cope in real time with high volume of traffic without affecting/impeding the traffic flow.
An example of an IPS was snort_inline.

Currently IPS often are integrated into UTM(Unified Threat Management) devices.

To wrap up, the IDS and the IPS, for traffic analysis purposes are similar(except the IPS sits into the packet flow).
The main difference is in the active response capabilities.
So, originally, an IDS, compared with an IPS, only detected/alerted/logged and did not block "unwanted" traffic.
Over the time the difference blurred and it is rather related to the response to threat capabilities, the IDS may let some malicious traffic to slip through before it can respond(with the help of a networking device), whereas an IPS is capable to respond immediately to the threat, -> the IDS response is rather reactive(subsequent traffic will be blocked) and the IPS' one is proactive.
Currently, which may add some confusion, some vendors allow their IPS solutions(which run in inline mode) to be used in promiscuous mode too, thus to act as a pure IDS(detect+alert+log capabilities) or as an IDS with blocking capabilities with the help of a firewall or a router(similar with what SnortSam does).

 1.2 IDS/IPS detection methods
It's somehow relative to describe the detections methods used by IDS/IPS solutions, terminology may vary from vendor to vendor.

NIST, in SP800-94, Guide to Intrusion detection and Intrusion Prevention Systems(IDPS)
http://csrc.nist.gov/publications/nistpubs/800-94/SP800-94.pdf
describes three main detection methodologies(although the definitions from there may not correlate with the current IDS/IPS market):
- signature-based detection
- anomaly-based detection
- stateful protocol analysis

Some vendors may include multiple detection methods within their IDS/IPS solutions.

 1.2.1 Signature-based detection
The signature-based detection(named by some rule based detection) is probably the most common detection method used by IDS/IPS.

IDS solutions originally were looking for suspicious activities such as port scanning(reconnaissance detection) or various login attempts(like root login over telnet or multiple login attempts).
The exploitation of applications became more and more predominant, signatures to detect the exploits of known vulnerabilities of various applications were added to IDS/IPS solutions. The IDS/IPS could detect/block attacks against OS, server applications, and client applications.

The detection capabilities of a signature-based IDS/IPS solutions relate to a few things.

Earlier, signature-based detection used to spot attacks based on (static) pattern matching(a string or a regular expression match against traffic), without really understanding protocols.
Currently, IDS/IPS solutions using signature-based detection may include protocol analyzers for this detection method, basically combining pattern matching with protocol analysis.
At the beginning pattern matching and protocol analysis were different approaches.
The pattern matching, for example, searched the entire HTTP packet(raw search, byte-by-byte) for a malicious URL.
The protocol analysis(dealing with formatted data of network traffic, unwrapping the layers of protocols(IP,TCP or application protocols(HTTP, SMTP),etc.) and inspecting them) only looked at the URL field of the HTTP request message for an unexpected value. When URL encoding was used as a form of obfuscation in an attempt to evade IDS/IPS solutions, the pattern matching approach had to start understand HTTP in order to locate the URL field and try to decode the value within, thus had to include some form of protocol analysis.
Although pattern matching is combined with protocol analysis(or vice-versa, the protocol analysis solutions does pattern matching at certain points in the application level decoding process), some IDS/IPS solutions might not fully(or truly) understand protocols, for example they understand (sort of) HTTP, but they are not native HTTP speakers -> this can lead to IDS/IPS evasion.
Also, the types of signatures used by IDS/IPS solutions can affect their capabilities. In order to detect/block exploits of vulnerabilities, IDS/IPS vendors can use exploit-based signatures(earlier generation of signatures) or/and vulnerability-based signatures(later generation of signatures).

 1.2.1.1 Exploit-based signatures
Such signatures focus on exploits of a vulnerability.
This means that for a given vulnerability, in order to write exploit-based signatures, the author of the signature must understand the (known) exploits and how they trigger the vulnerability.

The immediate weakness of such an approach is that the author may need to write multiple exploit-based signatures for a given known vulnerability and those signatures are able to detect just known exploits.
For example, a 0-day vulnerability is found and PoC(proof-of-concept code) used to exploit that vulnerability is publicly disclosed. An IDS/IPS vendor analyzes the PoC and writes an exploit-based signature to detect the exploit based on the PoC published.
If a new 0-day exploit for that specific vulnerability appears, the previous exploit-based signature will fail to detect this 0-day exploit, the vendor needing to write a new signature. Also small modifications to the original PoC exploit might cause the IDS/IPS to fail to detect the exploit.
Thus, the IDS/IPS solution is rather just a reactive one.

One thing to be noted here, the author of the signature may have good understanding of exploit techniques and relate those to the particular vulnerability when writing signatures, rather than just writing signatures for exploits that are publicly available for that particular vulnerability, thus the IDS/IPS might be able to detect variants of the exploits or new exploits that will be later published or used by attackers.

The exploit-based signatures are able to identify the exploit used in the attack and log it.

 1.2.1.2 Vulnerability-based signatures
Instead of focusing on the exploits, such signatures focus on the vulnerability itself.

For writing such a signature, the author must understand the vulnerability and the possible ways to exploit it.

The main advantage of vulnerability-based signatures is that they should be able to catch most exploits for a given known vulnerability, either variants of publicly available exploits or 0-days exploits.
It's more difficult to write vulnerability-based signatures because the vulnerability itself must be well understood and the author must have good understanding of exploit techniques. Also for a particular vulnerability it may be tricky to write a vulnerability-based signature due to the nature of the vulnerability(it may end up blocking legitimate traffic, causing false positives).

To be noted that for using vulnerability-based signatures the IDS/IPS vendor must include some form of protocol analysis within its IDS/IPS solution in order to avoid evasion. For example a certain client HTTP request containing a malicious URL can be used to trigger some vulnerability on a web server. Using URL encoding techniques(legal and illegal) attackers are able to have dozens, or possibly hundreds of exploit variants for that particular vulnerability, and the IDS/IPS must feature a good URL decoder to decode the URL in order to be able to identify the needed pattern that triggers the vulnerability.
Due to advanced understanding of protocols, protocol analysis-based solutions(doing pattern matching) may block exploits with a single signature that monitors the traffic for a specific vulnerability, and also have a low number of false positives.

In the context of vulnerability-based signatures, the term virtual patching is valid.

The vulnerability-based signatures aren't necessarily able to identify the exact exploit used in the attack.

 1.2.1.3 Vulnerability-based signatures vs 0-day vulnerabilities
As we saw vulnerability-based signatures are able (being quite good at) to detect/block known/0-day exploits of known vulnerabilities.
How about exploits of 0-day vulnerabilities?

In some occasion an IPS using vulnerability-based signatures might be able to block exploits against 0-day vulnerabilities.
For example writing a "generic" vulnerability-based signature for attempting to catch exploits of a specific type of vulnerability, say heap overflow vulnerability, may result, in certain circumstances, in the block of attempts to exploit a 0-day heap overflow vulnerability.

However, new classes of vulnerabilities would likely not be detected by signature-based IPS.

Also the IPS might not be able to detect the exploit of a vulnerability, but based on the nature of the action of the exploit, for example the attacker attempts arbitrary code execution like connect-back shellcode, the IPS might detect the shellcode payload and block the attacker's subsequent access to the compromised machine.

 1.2.1.4 Exploit-based signatures vs vulnerability-based signatures
To wrap up, ideally IDS/IPS vendors would use vulnerability-based signatures. This provides the admins with a virtual patching solution.

However, based on the nature of the vulnerability, this is not always possible or more difficult to achieve.
Without understanding the vulnerability, good vulnerability-based signatures are impossible to write.
As a quick aid workaround, an exploit-based signature might be written.
Note that sometimes the vendor won't be able to write even an exploit-signature. In this case the signature might be disabled by default(causing false positives).

Also, an IDS/IPS vendor might use both vulnerability-based signatures and exploit-based signatures, the exploit-based signature being able to identify the exact exploit used and log that.

Pay attention to the signature details, note if the vendor classifies or not the signature as exploit-based or vulnerability-based.

 1.2.2 Anomaly-based detection
Anomaly-based detection works by looking for traffic that deviates from traffic that it is considered normal. Profiles that represent the normal behavior are created, traffic detected outside the normal policy is blocked and/or an alert is issued.
Normal profiles can be created either manually or dynamically.

The anomaly-based detection is also named by some policy-based detection.
Some vendors may label their anomaly-based detection as heuristic or behavioral analysis.

The advantage of anomaly-based detection is that it can block exploits against 0-day vulnerabilities.

The downside with anomaly-based detection is the difficulty in creation of the normal profile, especially in large networks. Accidentally, malicious activity can be included within the normal profile on a live network.
If a too aggressive policy is used, the rate of false positives is high, thus such a profile can affect usability.

 1.2.3 Stateful protocol analysis detection
Stateful protocol analysis detection works by having IDS/IPS vendor defined profiles for protocols(like HTTP, SMTP) per protocol standards(such as RFCs) and observing deviations of traffic from these profiles.

These profiles might vary from vendor to vendor, some may tighten RFC standards or complement protocol standards to take into consideration proprietary features.

The main drawback is that defining profiles for protocols is rather a complex task, for example some standards are evasive in explaining certain details, which may lead to variations among implementations.

 1.2.4 Virtual patching using IPS
Virtual patching using an IPS can be a good short-term fix for certain vulnerabilities, or a long-term one if the application or OS is no longer being maintained(thus the vendor won't patch the vulnerability).

In the context of a signature-based IPS, virtual patching is essentially the process of putting in place an IPS signature set to block in order to prevent the exploitation of a vulnerability. Some IPS vendors, in addition of automatically creating such a signature, allow you to manually write signatures(useful in case you ran some custom application).
Once the vulnerability has been addressed by the vendor/developer and you've applied the patch in the production network, you can disable the signature, set it to detect only or leave it in block mode.
Virtual patching helps reducing the vulnerability window between vulnerability disclosure and patch deployment.
One thing to be noted, virtual patching cannot be truly related to exploit-based signatures, rather to vulnerability-based signatures.

IPS solutions can detect and block various attacks, such attacks on client/server side applications, on hosts, protocols, network attacks etc.

But due to the technical limitation of an IPS, using it for virtual patching in certain conditions may be inappropriate.
For example, an IPS can detect and block some attacks against web applications. WAFs(Web Application Firewalls) provide an application layer logic superior to IPS solutions.
Say, for a given URL, a WAF knows that authentication is required(and based on the client's User-Agent form-based authentication or basic authentication is expected), only certain HTTP methods(like GET/POST/HEAD) are accepted, high bits characters are not allowed, the URL is not allowed to contain more than 30 characters. The WAF understands HTML , XML, JavaScript or SQL and is able to cope with layer over layer of code obfuscation.
An IPS does buffering, however in the process can let some packets through without having the complete message, while an WAF does not do that in order to keep track of the application specific context of requests and responses, the application-awareness of the WAF being superior to the IPS one.
Relating this to TMG we would note that TMG can act as a reverse-proxy. Without being a full blown WAF, in reverse proxy mode, TMG's web server publishing rules can proactively block exploits of certain vulnerabilities better than the NIS does.
For example, simply using path restrictions on TMG's web server publishing rules can block the exploitation(leading to unauthorized data access) of the two vulnerabilities presented below within the NIS signature examples (exploit-based and vulnerability-based signatures) prior of knowing the two vulnerabilities. The logic within TMG's web publishing rules allow for a certain URL to have TMG pre-authenticate the user and then delegate the credentials to the back application, whitelist the URLs a specific user can access, along with the HTTP methods allowed, to name a few.
Having NIS as an IPS integrated within TMG, better protection for web applications can be achieved compared to a standalone IPS.

 1.3 Time frames(or exposure windows) related to a vulnerability and IPS
Let's attempt to detail a little how an IPS might help mitigate the exploit of a vulnerability understanding the exposure window.

- A vulnerability(named x) is discovered, either by a researcher or an attacker. Note that a vulnerability might also be discovered internally by the vendor.
- The researcher may first attempt to contact/inform the vendor about the vulnerability. At this time the vulnerability was (likely) not acknowledged in public, however it may exist the possibility that an attacker already discovered this vulnerability and actively exploits it.
- The attacker on the other side may attempt to exploit for profit the vulnerability by himself or sell it to others. Similarly, at this time the vulnerability was (likely) not acknowledged in public.
- After being contacted by the researcher, the vendor may acknowledge (in public) the existence of the vulnerability or downplay it. Yet exact details about the vulnerability and how to exploit it are unknown in public.
- The attacker(s), in case they discovered the vulnerability, actively exploit it. Yet details about the vulnerability and how to exploit it are unknown in public.
- Instead of contacting the vendor or due to the vulnerability being downplayed or a patch delayed too much by the vendor, the researcher makes public the vulnerability, likely along with PoC. At this time the world(researchers, vendors, users, attackers, etc) becomes aware of the vulnerability. Note that discovery not equal disclosure.
- In case being discovered by attackers who actively exploit it(the existence of the vulnerability being unknown to the public), for example, by investigating a breached network, the exploit of the vulnerability x might be found. At this time the public/vendor may become aware of the vulnerability(it may take a while for the vendor to acknowledge the vulnerability), however exact details about the vulnerability and how to exploit it may or may not be publicly released.
- So depending how a vulnerability is discovered and/or reported to the vendor, exact details about the vulnerability and how to exploit it may or may not be publicly released. However IPS vendors might get these details and release signatures addressing this vulnerability even when the details are not publicly disclosed, assuming they become aware of the vulnerability. Or, depending on the detection methods(say heuristic detection) and capabilities of the IPS, the IPS solution may already block (some) exploits of the vulnerability x.
- If exact details about the vulnerability and how to exploit it are publicly released, soon variants of exploits might start to circulate in public(either commercial ones or ones posted on full disclosure sites). IPS(using signature-based detection) vendors by now should have signatures addressing this vulnerability.
- The vendor issues a patch fixing the vulnerability or the vendor may not ever issue a patch, either ignoring the vulnerability or due to the vulnerability being found on an unsupported OS or software version(although still used in production by some).
- In case the exact details about the vulnerability and how to exploit it are not publicly released, attackers have a small opportunity window in trying to reverse engineer the patch in order to get more details about the vulnerability and how to exploit it.
- Till the patch is tested in order to be deployed in production, an IPS signature can be used to reduce the attack surface on vulnerable hosts/applications/etc.
- The patch is finally deployed in production closing the exposure window. This will be true on some networks; others will delay it due to various reasons, remaining exposed.
- A patch will never be provided by the vendor, customers either fixing the vulnerability or using workarounds on their own. In this case an IPS signature might be used to virtual patch for indefinite time the vulnerability.

 2. Where TMG's NIS stands
After we have quickly navigated through the evolution of IDS/IPS, let's take a look at TMG's NIS.

Currently NIS is integrated within Forefront TMG 2010 as a component of its Intrusion Prevention System (IPS) solution.

NIS works inline, and it is based on the Generic Application-level Protocol Analyzer (GAPA) research.
http://research.microsoft.com/apps/pubs/default.aspx?id=70223
|From this paper is interesting to note Microsoft's explanation on how they can detect the CodeRed worm with GAPAL(relate this to our discussion regarding pattern matching and protocols analysis):
"The HTTP protocol identifies the URLs in the HTTP requests and passes them on to the URL parser using the layering mechanism. The URL parser looks for URLs interpreted by the IDA ISAPI filter and breaks them into their constituent components. The analysis handler then checks whether the buffer parameter exceeds a certain length (causing a buffer overflow), and raises an alert when it does."

NIS is a signature-based network Intrusion Prevention System (IPS), combines(or seems to combine) protocol analysis with pattern matching. NIS, as writing, supports the HTTP, DNS, SMB, SMB2, NetBIOS, MSRPC, SMTP, POP3, IMAP, MIME protocols.
Currently NIS offers protection against network based attacks and against some Web based attacks, but doesn't provide protection against exploit files (file based attacks).

NIS uses both exploit-based signatures and vulnerability-based signatures.

Also NIS offers a protocol anomaly detection functionality, monitoring the network traffic to be in compliance with protocol standards such as RFCs and common implementations.

Right now NIS is able to block known and unknown exploits of known vulnerabilities.
As the time of writing, NIS does not block exploits of unknown vulnerabilities, it is rather focused on reducing the vulnerability window between vulnerability disclosure and patch deployment.

Some encrypted traffic, such as HTTPS traffic(either when TMG acts as a reverse-proxy or forward proxy) can be inspected by NIS.

Speaking about packet buffering, within the GAPA research paper, it is mentioned:
http://research.microsoft.com/apps/pubs/default.aspx?id=70223
"We buffer a packet in the engine only as long as its fields are being parsed by the grammar parser, and release it to the application immediately afterwards. This may cause potentially malicious packets to the application, however, these packets will be incomplete. The security assumption we are making is that if the GAPA engine does not have enough of the message to make a decision about whether it is malicious or not, the partial content is not yet dangerous enough to cause errors in the application."

 2.1 NIS signature examples
To put into practice all of the above theories and see the differences between an exploit-based signature and a vulnerability-based signature, let's look at some TMG NIS signatures, and exercise them with PoC and variations of the publicly available PoC.

In order for everybody to be able to follow without much troubles, we will discuss some relatively easy to understand vulnerabilities, and use common tools(like WFetch which is rather an admin tool) to "exploit" them.

The signature set version on my TMG 2010 is 9.61.

 2.1.1 A look at a NIS exploit-based signature
Let's analyze Expl:Win/IISUnicode.WebDav.PE!2009-1535 (enabled and set to block by default).

More details about this signature can be found at:
http://www.microsoft.com/security/portal/Threat/Encyclopedia/NIS.aspx?threat=Expl-Win-IISUnicode-WebDav-PE-2009-1535

The vulnerability(IIS 5.1 and 6.0 WebDAV Authentication Bypass Vulnerability - CVE-2009-1535) is documented within MS09-020:
http://www.microsoft.com/technet/security/Bulletin/MS09-020.mspx

This vulnerability was publicly disclosed and PoC can be found at:
http://archives.neohapsis.com/archives/fulldisclosure/2009-05/att-0135/IIS_Advisory.pdf

IIS 5.1/IIS 6.0 are affected by this vulnerability.
An attacker could exploit this vulnerability by creating a specially crafted anonymous HTTP request to gain access to a location that typically requires authentication.
The original published PoC inserts the %c0%af unicode sequence within the name of a protected folder requested within the malicious URL. Also a "Translate: f" HTTP header is used within the malicious HTTP request in order to instruct the server to handle the request using the WebDav extension. This leads to authentication bypass due to the fact that the WebDAV extension does not properly decode the crafted requested URL(WebDav must be enabled on the web server).
The %c0%af unicode sequence is a forward-slash(/) that's been converted to UTF-8 Unicode using double-byte encoding(overlong encoding method with a size of 2), see the vulnerability-based signature example from below for more details about this encoding method.
The Translate header with a value of f, prompts the WebDav server to return the actual source of the file(this is useful for example to retrieve the actual source of an ASP page rather than the processed HTML file, see http://msdn.microsoft.com/en-us/library/cc250063(v=prot.13).aspx).
To exemplify this, I have an IIS 5.1 server(192.168.22.158), WebDav is enabled on that server (it is enabled by default on IIS 5.1).
A file a.txt is found inside the share folder on this server.
Authentication is required for accessing that file.

First we will attempt to bypass the authentication exploiting the WebDAV Authentication Bypass vulnerability without TMG on the path.
Then we will put TMG 2010 on the path, with the NIS enabled, default settings. For simplicity, we will not use a web server publishing rule(to avoid the protection offered by such a rule), just an access rule, simulating a client behind TMG that tries to access an external server.

Without TMG on the path, let's attempt to get access to the a.txt file.
Our crafted request:
http://192.168.22.158/s%c0%afhare/a.txt
Submit this request using WFetch, and as can be seen we were able to access that file:

Now let's try to do this with TMG on the path, TMG's NIS kicks in and blocks the request:

The associated log from TMG to our crafted request, as can be seen the signature Vuln:Win/IIS.URL.PE!2010-2731 detected the exploit(also we can see from the log the malicious URL used):

 2.1.1.1 Attempt to bypass the exploit-based signature(alternative publicly available PoC 1)
Another PoC(different sequence) can be found here:
http://www.breakingpointsystems.com/community/blog/slash-and-burn-the-iis-60-webdav-bug/

Submit the request with WFetch and TMG's NIS blocks the request(I'm not sure if this is on purpose or by accident, this URL contains the %c0%af sequence):

The associated log from TMG to our crafted request, as can be seen the signature Vuln:Win/IIS.URL.PE!2010-2731 detected the exploit:

 2.1.1.2 Attempt to bypass the exploit-based signature(alternative publicly available PoC 2)
Myself I've detailed another sequence that works here:
http://www.carbonwind.net/blog/post/Forefront-TMG-Beta-3-NIS-e28093-Signature-Details-Properties.aspx

Submit the request and TMG's NIS fails to detect the exploit:

The associated log from TMG to our crafted request, as can be seen the request passed through undetected by the IPS:

That's the hassle with the exploit-based signatures.

And as you may have observed one can't use such a signature for virtual patching, the server still being exposed to public available PoC.

To note that in order to catch all the possible exploits(seems that triple-byte encoding can be used too) with an IPS signature is kind of tricky, many workable sequences are possible. One may attempt to block the use of high-bit characters within an URL.

 2.1.2 A look at a NIS vulnerability-based signature
Let's analyze Vuln:Win/IIS.URL.PE!2010-2731(enabled and set to block by default).

More details about this signature can be found at:
http://www.microsoft.com/security/portal/Threat/Encyclopedia/NIS.aspx?threat=Vuln-Win-IIS-URL-PE-2010-2731

The vulnerability(Directory Authentication Bypass Vulnerability - CVE-2010-2731) is documented within MS10-065:
http://www.microsoft.com/technet/security/bulletin/MS10-065.mspx

This vulnerability was publicly disclosed and PoC can be found at:
http://soroush.secproject.com/blog/2010/07/iis5-1-directory-authentication-bypass-by-using-i30index_allocation/

IIS 5.1(Windows XP SP 3) is affected by this vulnerability.
An attacker could send a specially crafted request to a URL that requires authentication to bypass authentication and execute ASP script to which the attacker should not have access.
Specifically by appending the :$i30:$INDEX_ALLOCATION string to the protected folder(when directory-based basic authentication is enabled) within the malicious URL, authentication will be bypassed.

To exemplify this, I have an IIS 5.1 server(192.168.22.158).
A file c.asp is found inside the a folder on this server.
Authentication is required for accessing that file.

First we will attempt to bypass the authentication exploiting the Directory Authentication Bypass vulnerability without TMG on the path.
Then we will put TMG 2010 on the path, with the NIS enabled, default settings. For simplicity, we will not use a web server publishing rule(to avoid the protection offered by such a rule), just an access rule, simulating a client behind TMG that tries to access an external server.

Without TMG on the path, let's attempt to get access to the c.asp file, success:

Now let's try to do this with TMG on the path, TMG's NIS kicks in and blocks the request:

The associated log from TMG to our crafted request, as can be seen the signature Vuln:Win/IIS.URL.PE!2010-2731 detected the exploit:

 2.1.2.1 Attempts to bypass the vulnerability-based signature
Let's attempt to see if we can evade the Vuln:Win/IIS.URL.PE!2010-2731 NIS signature.

At a glance it may appear that we can successfully block the attack by using a pattern matching for the requested URL.
For example one may try to use TMG's HTTP filter and block the :$i30:$INDEX_ALLOCATION pattern or some suggestive parts of it(you can try this and see what happens). Note that the HTTP filter understands some URL encoding formats.

As we mentioned, attackers can use various URL encoding methods(legal or illegal) in an attempt to bypass an IPS.
Below we will attempt a few of them.

You should test first without TMG on the path, and see if the URL encoding "trick" works and you can successfully access the needed file.
Some servers may not process the crafted URL(this varies per platform(IIS vs Apache), or per various server versions). We "deal" below with IIS 5.1.

 2.1.2.1.1 Using URL encoding for special characters
First using valid URL encoding, each non-ASCII characters (all characters that require URL encoding -> a superset of non ASCII characters) are URL-encoded.
For more info of such characters see:
http://www.w3schools.com/TAGS/ref_urlencode.asp

We will use URL encoding for the $ and : characters.
$ encoded becomes %24
: encoded becomes %3A

So our request now looks like:
http://192.168.22.158/a%3A%24i30%3A%24INDEX_ALLOCATION/c.asp

Submit it using WFetch:

Note TMG's response, NIS blocked our request and Vuln:Win/IIS.URL.PE!2010-2731 NIS signature successfully detected the exploit. Also from TMG's log we can see the actual exploit used, since the request URL is logged.

To be noted that we cannot bypass like so a TMG HTTP filter signature set to block a request URL containing :$i30:$INDEX_ALLOCATION.

 2.1.2.1.2 Using hex encoding for a random character
For example let's take the O character.
O hex encoded becomes %4f.

Our request now is:
http://192.168.22.158/a:$i30:$INDEX_ALL%4fCATION/c.asp

Submit this request using WFecth:

Once again we cannot bypass the Vuln:Win/IIS.URL.PE!2010-2731 NIS signature.

And one more time to be noted that we cannot bypass like so the HTTP filter signature set to block a request URL containing :$i30:$INDEX_ALLOCATION.

 2.1.2.1.3 Using double-byte encoding for a random character
We will continue "working" with the O character.
We will use the double-byte encoding process(UTF-8 Unicode with the overlong encoding method with a size of 2, some name it short UTF-8 encoding). To not confuse it with double encoding.

The rules for UTF-8 encoding can be found at:
http://www.ietf.org/rfc/rfc2279.txt

A little bit more info can be found at:
http://www.owasp.org/index.php?title=Canonicalization,_locale_and_Unicode&setlang=es
http://www.cl.cam.ac.uk/~mgk25/unicode.html

If you would like to skip the "technical" details, you can use a tool like HttpChameleon, see below.
If you would like to manually test it yourself, here is a simplistic explanation.

RFC2279 says: "Determine the number of octets required from the character value".
We can hex encode O as %4f, we only need a byte for that. The binary representation for O is 01001111.
According to RFC2279, for 1 byte we have 0xxxxxxx(which fits 01001111).

However, we can try to use two bytes for encoding O.
For two bytes, according to RFC2279 we have 110xxxxx 10xxxxxx.

We can substitute the x's for the two byte encoding, with bits of the character value, according to RFC2279 "starting from the lower-order bits of the character value and putting them first in the last octet of the sequence, then the next to last, etc. until all x bits are filled in."

To make it more easy to visualize the process, I've split binary O(single byte) in two parts, a 2 bits one and a 6 bits second one, and you can note where those bits went. The remaining x bits were padded with 0(we padded three x with 0).

So O became 11000001 10001111. Which transformed into hex gives us: %c1%8f.

Alternatively you can do it the easy way, using a GUI and a few mouse clicks, with HttpChameleon.

Now our request looks like:
http://192.168.22.158/a:$i30:$INDEX_ALL%c1%8fCATION/c.asp

We can submit it using WFetch:

And we could not bypass the Vuln:Win/IIS.URL.PE!2010-2731 NIS signature.

Note that we can bypass like so the HTTP filter signature set to block a request URL containing :$i30:$INDEX_ALLOCATION.
Additionally we can attempt to double-byte encode various characters from the :$i30:$INDEX_ALLOCATION. Or encode a combination of characters.
That's why we need the IPS to be able to understand such encoding methods in order to write a vulnerability-based signature, it's not enough for the IPS to find the URL field from the HTTP request and do a pattern matching on its value.

In case you are curios, I've double-byte encoded the entire URL and NIS detected this request:

 2.1.2.1.4 Using triple-byte encoding for a random character
To be consistent, we will use the O character again.

The triple-byte encoding process(UTF-8 Unicode with the overlong encoding method with a size of 3, some name it long UTF-8 encoding) is similar with the two-bytes one, so we will not explain it below. Just show the table helping you to visualize the process.

So now our request looks like:
http://192.168.22.158/a:$i30:$INDEX_ALL%e0%81%8fCATION/c.asp

Submit this request using WFetch:

And again we cannot bypass TMG's NIS:

Note that we can bypass like so the HTTP filter signature set to block a request URL containing :$i30:$INDEX_ALLOCATION.

In case you are curios, I've triple-byte encoded the entire URL and NIS detected this request:

 2.1.2.2 A few thoughts
Although at the surface it seems fairly trivial to block an attack attempting to exploit the IIS 5.1 Directory Authentication Bypass vulnerability by using a pattern matching for the request URL, due to the various URL encoding methods(combined with random character encoding) that can be used to craft a malicious request, simply writing an exploit-based IPS signature for the original PoC would just offer a false sense of security.

That's why in the context of a vulnerability-based signature we can talk about virtual patching, until we wait for a patch from the vendor, we can use such an IPS signature to provide a virtual patch that would block most of the exploits.

 2.3 A look at a NIS policy-based signature
Microsoft says(source: NIS in TMG Whitepaper,
http://download.microsoft.com/download/f/4/0/f40887fd-648b-40e1-b79b-aae43cedca4c/nis%20in%20tmg%20whitepaper.docx
that such "signatures are generally used for auditing purposes and are developed when neither vulnerability nor an exploit-based signature can be written".

Note that currently, by default, the policy-based signatures are disabled:
http://blogs.technet.com/b/isablog/archive/2010/11/30/nis-signature-types-or-why-some-signatures-are-disabled-by-default.aspx

Such signatures may trigger false positives(you should read the signature description to see if there are any known false positives).

I suppose such signatures can also be used to monitor the use of certain application, like Live Messenger, Yahoo Messenger, Skype, etc., although currently there aren't such ones on TMG.

For example the Plcy:Win/SMB.WINREG.InfoDisc!2002-0049 signature, can be triggered when a user attempts to use the Microsoft Registry Editor(regedit) to edit registry entries on a remote computer through TMG.
http://www.microsoft.com/security/portal/Threat/Encyclopedia/NIS.aspx?threat=Plcy-Win-SMB-WINREG-InfoDisc-2002-0049

If we enable this signature, and set it to block like below:

And then we attempt to edit registry entries on a remote computer through TMG(below from 192.168.10.2 accessing Windows registry on 192.168.22.158, TMG provides network separation between the 192.168.10.0/24 and 192.168.22.0/24 networks), TMG's NIS will block the request.

Obviously we did not attempt to exploit the MS02-003 vulnerability, this type of false positive is acknowledged within the signature description: "This signature will trigger on any traffic that attempts to connect to WinReg remotely." http://www.microsoft.com/security/portal/Threat/Encyclopedia/NIS.aspx?threat=Plcy-Win-SMB-WINREG-InfoDisc-2002-0049

 2.4 Protocol anomaly detection examples
In order to test the protocol anomaly detection functionality, first we have to enable NIS to block protocol anomalies(by default is set to allow):

Probably the simplest way we can trigger NIS to block a protocol anomaly is by using a client request with a HTTP version 1.2(currently the latest version of HTTP is 1.1, version 1.2 is an invalid HTTP version within a client request).

We can easily do that with WFetch, note TMG's reply:

From TMG's log we cannot see what anomaly caused the IPS to deny our request, just that a HTTP request was denied:

Also an alert was raised on TMG for the protocol anomaly blocked:

Another simple way to make NIS protocol anomaly detection to kick in is when a HTTP header is being terminated improperly; normally it must end with CRLF ("\r\n", Carriage Return and Line Feed).
Instead, using WFetch we can terminate it with "\rn":

Another simple(and funny) test, say I attempt to misuse the CIFS protocol(rather the TCP port 445) and instead setup a covert HTTP channel(attackers might attempt to setup covert channels over commonly allowed protocols through firewalls, say in this case TMG provides network separation for some internal networks).
Note the request being denied by NIS due to a protocol anomaly:

 3. Conclusion
In this article we've quickly navigated through the history of IDS/IPS solutions, described TMG's NIS detection methods, gave examples of NIS signatures for some vulnerabilities and examples of protocol anomalies detected/blocked by TMG's NIS.