09.04.2008
ISA Server 2006 - IPsec Tunnel Mode Site-to-Site VPN Connections: A Couple of Things That Are Not Supported

 - 1. Lack of support for Diffie-Hellman MODP Group 5 (1536-bit)
 - 2. Lack of support for AES
 - 3. The local subnet issue
 - 4. IKE authentication with certificates issue
 - 5. No IPComp
 - 6. The overlapping subnets issue(And no advanced NAT policies)

Using L2TP/IPsec for a VPN site-to-site connection is recommended when both VPN gateways are ISA 2004/2006 Firewalls. This is specified in clear in many docs on Microsoft's web site.

IPsec Tunnel Mode site-to-site VPN connections are used when interoperability with a VPN gateway from another vendor is required. ISA Server 2006 has passed the VPNC Basic Interoperability Test. If you scroll this document, you will find against which VPN gateways ISA Server 2006 was tested.

If you need to create an IPsec Tunnel Mode site-to-site VPN connection with a VPN gateway from another vendor, you may ran into a couple of unsupported settings or scenarios.
Let's point out some of them.

 1. Lack of support for Diffie-Hellman MODP Group 5 (1536-bit)
A easy to spot one will be the lack of support for Diffie-Hellman MODP Group 5 (1536-bit). Popular VPN gateways often use this group. ISA Server 2006 can use instead the stronger Diffie-Hellman MODP Group 14, see Figure1. Actually, Diffie-Hellman MODP Group 14 matches the strength of 3DES. Unfortunetely, some vendors are "stuck" at the Diffie-Hellman MODP Group 5 "level", and they do not support Diffie-Hellman MODP Group 14. You may like to search the VPNC Members Supported Features - IPsec to see which vendor listed there has support for Diffie-Hellman MODP Group 14.

Figure1: Diffie-Hellman Groups

 2. Lack of support for AES
Another easy to spot one will be the lack of support for AES. AES is now widely available on popular VPN gateways.
ISA Server 2006 only supports DES and 3DES, see Figure2.
Although AES is present on popular VPN gateways, there is a hint: this does not automatically mean that you can benefit from its strength. The derivation of AES keys requires the use of algorithms of certain strength. For example, since above we've talked about Diffie-Hellman Groups, AES with 128 bits keys requires for comparable strength, Diffie-Hellman MODP Group 15 (3072-bit). This may not be very practical (too slow lowering system's performance).
Still, even when for example, Diffie-Hellman MODP Group 5 is used for keys derivation, AES would be the preferred choiced over 3DES due to efficiency reasons.

Figure2: Symmetric Encryption Algorithms

 3. The local subnet issue
The one that follows is not that easy to discover, but if you are in the "right" place you will definetelly have headaches.
In certain situations, you may need to specify that the local site IP addresses range to include only a subnet or just a single IP address from the internal network.
For example, the local office uses the 192.168.40.0/24 network and the remote office uses the 192.168.30.0/24 network.
You just need to specify as the local network a single host, 192.168.40.2 and for the remote network, again just a single host, 192.168.30.2 for the site-to-site VPN connection.
However it is not possible to specify from ISA's GUI that just a single host from the internal network will be used for the VPN site-to-site connection.

Let's visualize this a little bit. One endpoint of the VPN tunnel uses an ISA Server 2006 Server installed on Windows 2003 Standard R2 SP2 and the other endpoint of the VPN tunnel uses a Cisco router.
I've run the VPN site-to-site wizard, and skipped to add the network rule and the access rule. For the remote network I've specified only the 192.168.30.2 IP address. No problem with that.
But the VPN site-to-site wizard does not ask me anything about the local range of IP addresses that are to be used.
If I take a look at the site-to-site summary, I can notice that, although I did not entered anything, a local network appears to be configured (includes all the IP addresses used on ISA's Internal Network) - this ISA Server has only two NICs, internal and external -.

Figure3: IPsec Tunnel Mode VPN site-to-site Summary - No Network Rule

Even more, if I take a look at the Quick Mode Filters, I can spot that the proxy identities that are to be used during IKE Quick Mode negotiations have been defined, see Figure4 and Figure5. You can view this filters either using the "IP Security Monitor" snap-in or "netsh" commands.

Figure4: IP Security Monitor Quick Mode Generic Filters

Figure5: netsh Quick Mode Generic Filters

Obviously the proxy identities are wrong, instead of 192.168.40.0/24, I want to use 192.168.40.2/32.
And, as you might know, if the proxy identities do not match at both endpoints of the VPN tunnels, IKE Phase II (QM) negotiations will fail, thus the "tunnel" will not be established.

You may think: "what about the required network rule, I will define a network rule between the remote site and this local host(192.168.40.2), and maybe things will change".
Well, it does not work like so. I've created a Computer Object for this local machine, seeFigure6.

Figure6: Computer Object

And a network rule with a route relationship between the remote site and the local computer.

Figure7: Network Rule

If we look now at the site-to-site summary (see Figure8), the local network still includes the entire network 192.168.40.0/24. Only 192.168.40.2/32 is shown as a routable local IP address. And, I'm suggested that on the other end of the tunnel, to use 192.168.40.2/32 as the remote network.

Figure8: IPsec Tunnel Mode VPN site-to-site New Summary

Analyze again the Quick Mode filters and nothing has changed, still the wrong proxy identities are used, see Figure9.

Figure9: IP Security Monitor Quick Mode Same Generic Filters

Moving at the other endpoint of the VPN tunnel, and looking at the proxy identities defined on the Cisco router, we can see they look fine.

Figure10: Cisco Router IPsec Tunnel Mode VPN Expected Proxy Identities

Although we know that IKE QM negotiations will fail, let's play the game and try to bring the tunnel up. I will initiate the tunnel from ISA's side, by pinging from host 192.168.40.2 to host 192.168.30.2.

I've enabled "crypto ipsec" debugging on the Cisco router. As expected, IKE QM will fail. InFigure 11 we can see the error shown on a Cisco 3620 router(IOS version 12.3-16), "proxy identities not supported", and in Figure 12, the error shown on a Cisco 3640 router (IOS version 12.4-7), "no IPSEC cryptomap exists" - note that the error from Figure 12 was shown too on a Cisco 3725 router (IOS version 12.4-3).
We can clearly see that the proxy identities proposed by ISA were not the ones expected by the Cisco router.

Figure11: Cisco 3620 Router (IOS version 12.3-16): "proxy identities not supported"

Figure12: Cisco 3640 Router (IOS version 12.4-7): "no IPSEC cryptomap exists"

If we analize the Oakley.log from ISA, in case of the VPN site-to-site connection with the Cisco 3620 router, we can notice that IKE Main Mode negotiations were completed, and then ISA attempted to negotiate IKE QM with the wrong proxy identities, see Figure13. Thus, the Cisco router informs ISA that it has not accepted its proposal (NO-PROPOSAL-CHOSEN, see RFC2408 Internet Security Association and Key Management Protocol (ISAKMP), an "Internet Official Protocol Standards" (STD 1), for more details about Notify Messages Error - Types), see Figure14.

Figure13: ISA -> Cisco 3620 Router Oakley.log: Wrong Proxy IDs

Figure14: ISA -> Cisco 3620 Router Oakley.log: NO-PROPOSAL-CHOSEN

Same story with the Oakley.log from ISA in case of the VPN site-to-site connection with the Cisco 3640 router. See Figure15 and Figure16.

Figure15: ISA -> Cisco 3640 Router Oakley.log: Wrong Proxy IDs

Figure16: ISA -> Cisco 3640 Router Oakley.log: NO-PROPOSAL-CHOSEN

Interesting, if instead of trying to initiate the tunnel from ISA's side, I will initiate the tunnel from the Cisco router's side (by pinging from host 192.168.30.2 to host 192.168.40.2), the connection will be established.
Now the Cisco router starts the IKE QM negotiations and proposes the correct proxy IDs, ISA "senses" that its policy is "too general", adds on the fly new QM filters and accepts Cisco router's proposal. See Figure17.

By the way, if you post your Oakley.log on a forum searching for help when you have a problem with your VPN connection, and you need to remove your public IP addresses from the Oakley.log for privacy issues, make sure you do not miss their hex values too, see Figure17, InTunnelEndpt(192.168.22.240 here) and OutTunnelEndpt (192.168.22.222 here).

Figure17: CiscoRouter-> ISA Oakley.log: Policy Too General

If we look now at the Quick Mode filters, we can notice the newly added ones (correct ones), see Figure18.

Figure18: IP Security Monitor Quick Mode New Generic Filters

Obviously, the "result" cannot be considered a solution, there is no gambling here, the tunnel must be intiated from any direction and must stay up. Amazing how, although ISA Server 2006 features a powerful GUI which greatly simplifies the creation of VPN site-to-site connections, for something plain simple, we have a Las Vegas story.

 4. IKE authentication with certificates issue
ISA Server 2006 is capable of using certificates for authentication in case of IPsec Tunnel Mode site-to-site VPN connections. However there is a problem: you can specify only from which CA will accept ISA certificates, see Figure19.
This means that ISA will not perform additional checks over the certificate received from its IPsec peer (say check the name from the certificate). Therefore, the posibility that somebody in possesion of a valid certificate issued by the trusted CA, to establish the VPN connection, exists, because ISA cannot verify if indeed that certificate was issued to the expected VPN peer (if ISA could check the name from the certificate, this situation would be avoided, assuming that no one can obtain a certificate containing the correct name under a false identity from the CA).
Notice the big exclamation mark from Figure19 , which informs you to use a privately issued certificate (from a private certification authority, maybe your local private CA). If you click on the help link you will find more details about certificates and VPN connections. So it's always a good idea to read properly the documentation first. If you do so, you would be advised about the above described situation.

Figure19: IPsec Tunnel Mode VPN Site-to-Site Connection: Authentication with Certificates

 5. No IPComp
If you want to use compression with IPsec Tunnel Mode VPN Site-to-Site Connections you cannot do so. With other VPN gateways, compression occurs through IPComp (before encryption). ISA Server 2006 does not support IPComp.
If you want compression with VPN Site-to-Site Connections, you must use L2TP/IPsec site-to-site VPN connections (MPPC is used).

 6. The overlapping subnets issue (And no advanced NAT policies)
Another situation that may appear will be with overlapping subnets. Say, the local office uses 192.168.50.0/24 on its internal network, and so does the remote office. This can occur when you create a VPN site-to-site to a partner's office or so. Since both offices use IP addresses from private IP space, and they do not belong to the same company, there is a chance for subnets to overlap. A solution to this will be to apply policy based NAT(or whatever will be the marketing term) over VPN traffic, so hosts on one side can communicate with hosts on the other side using mapped IP addresses. Say, on the local VPN gateway, local network 192.168.50.0/24 will be NATed as 192.168.100.0/24 and the remote network will be 192.168.200.0/24. On the remote VPN gateway, local network 192.168.50.0/24 will be NATed as 192.168.200.0/24 and the remote network will be 192.168.100.0/24. Stating mappings(static NAT) can be defined for servers, say for a local server 192.168.50.10 -> 192.168.100.10. So when a remote host (192.168.50.100) needs to access our local server 192.168.50.10, it will initiate communications with the 192.168.100.10 host.
Unfortunetely, ISA does not have such flexibility regarding the NAT process, so we cannot do these. ISA's design was focused on advanced firewall capabilities rather than on advanced NAT policies, which are more common to routers.