Back in the ISA Server days the usage of SSL was very much related to the reverse web proxy scenario [1] where ISA was doing HTTPS bridging(additional usage was EAP-TLS).
Nowadays Forefront TMG(2010 version) makes way more use of SSL for its native features. Apart of the reverse web proxy or EAP-TLS, SSL is used with the outbound HTTPS Inspection feature, with SSTP and with the URL Filtering feature(to query the Microsoft Reputation Service); (also the email protection feature, depending on its configuration, may make use of SSL/TLS [13]; not discussed below).
All of these features make use of the Windows Schannel.
As writing, TMG is supported on Windows Server 2008 SP2 (64-bit) and Windows Server 2008 R2. [2]
There are a couple of differences between the Schannel versions of the mentioned Windows OS in terms of features(do note that with Windows Server 2008 Schannel was greatly improved compared to Windows Server 2003 in terms of features offered [5][12]).
Schannel on Windows Server 2008 R2 supports TLS 1.1 and TLS 1.2, however support for them is disabled by default. [3]
Schannel on Windows Server 2008 R2 has not enabled by default support for secp521r1 (aka NIST P-521) ECC curve. [4]
SSL 2.0 support
Do you need SSL 2.0 ?
No, actually SSL 2.0 usage is prohibited. [6]
On Forefront TMG 2010, as writing, meaningless of the OS version, SSL 2.0 is enabled on the server side(due to Windows 2008 server side Schannel settings): so the reverse web proxy and the SSTP listener will accept SSL 2.0 connections from clients.
Also for the client side, the outbound HTTPS Inspection feature uses a SSL 2.0 compatible TLS 1.0 Client Hello, incorporating SSL 2.0 cipher suites(it “forces” the use of it, by default Schannel disables, unless specifically requested by the application, support for SSL 2.0 on the client side).
For the client side, Windows registry, under:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\
the Client key is already added, with a DisabledByDefault setting set to 1.
As said, this value does not prevent an application to specifically use SSL 2.0 for its client side.
If you want that, delete the DisabledByDefault setting, and add the Enabled setting with a value of 0.
Note that you just removed like so any SSL 2.0 cipher suites from the outbound HTTPS Inspection‘s SSL 2.0 compatible Client Hello, but not disabled that type of Client Hello, I don’t know how to disable it unless you put TMG in FIPS mode.
If you want to disable SSL 2.0 support for the server side, Windows registry, under:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\
add the Server key.
Same settings DisabledByDefault and Enabled are available here too, set the appropriate values for them(like above for the client side).
Note that you just removed like so any SSL 2.0 cipher suites to be selected by the server side and to reject pure SSL 2.0 Client Hellos, but not from accepting SSL 2.0 compatible Client Hellos(like TLS 1.0 ones).
SSL 3.0 support
Do you need SSL 3.0 ?
Yes, you do, mostly if you need to support IE6 browsers. [7]
You may not need it if you want TMG to be in FIPS mode.
SSL 3.0 is enabled by default either on client or server side.
For the client and server side, Windows registry, under:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Client
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Server
the settings DisabledByDefault and Enabled are available if you want to disable support for SSL 3.0.
TLS 1.0 support
Do you need TLS 1.0 ?
Obviously you do.
TLS 1.0 is enabled by default either on client or server side.
The URL Filtering feature, to query the Microsoft Reputation Service, uses a TLS 1.0 Client Hello.
For the client and server side, Windows registry, under:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server
the settings DisabledByDefault and Enabled are available if you want to disable support for TLS 1.0.
TLS 1.1 and TLS 1.2 support
Do you need TLS 1.1 and TLS 1.2 ?
Not particularly, although TLS 1.2 is more secure than TLS 1.0.
On the browser side, only Opera currently enables support for TLS 1.1 and TLS 1.2 by default; IE8 and IE9 on Windows 7 support them but they are not enabled by default.
On the server side, some servers may enable support for TLS 1.1, but TLS 1.2 support in practice is quite inexistent [8]. Apache mod_gnutls and IIS 7.5 have support for TLS 1.1 and TLS 1.2; IIS 7.5 does not enable them by default.
Actually some web sites(including some Microsoft domains) reject Client Hellos messages advertising a TLS version higher than TLS 1.0, so, unless the outbound HTTPS Inspection has a fallback mechanism, if trying to use either TLS 1.1 or TLS 1.2 Client Hellos with such a site, will fail.
For the client and server side, Windows registry, under:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Server
and
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Server
if you add the DisabledByDefault setting with a value of 0, you will enable TLS 1.1 and TLS 1.2 support for:
- the SSTP listener
- the reverse web proxy: listens for TLS 1.1 and TLS 1.2 connections from clients and can connect to the published server with TLS 1.1 or TLS 1.2.
The outbound HTTPS Inspection will still use a SSL 2.0 compatible TLS 1.0 Client Hello.
Additional settings
You can put TMG in FIPS mode by using the System cryptography: Use FIPS 140 compliant cryptographic algorithms, including encryption, hashing and signing algorithms group policy; policy found under Local Computer->Windows Settings->Security Settings->Local Policies->Security Options. [9]
This would disable support for SSL 2.0, SSL 3.0, MD5 and RC4.
The outbound HTTPS Inspection will not use anymore the SSL 2.0 compatible TLS 1.0 Client Hello, instead it will use a TLS 1.0 Client Hello. In case of this TLS 1.0 Client Hello, the outbound HTTPS Inspection will use the SNI extension for connections requested by web proxy clients.
The server side on TMG will still accept SSL 2.0 compatible Client Hellos.
You can control the preferred cipher suites order and remove support for some cipher suites(like SSL 2.0 ones) using another group policy, the SSL Cipher Suite Order; policy found under Local Computer->Computer Configuration->Administrative Templates->Network->SSL Configuration Settings. [10][11]
Note that in case of a group policy, the domain one should take precedence over the local one.
Also all the enabled by default cipher suites are listed under:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Cryptography\Configuration\Local\SSL\00010002\ within the Functions setting.
If you don’t use the local group policy you can set/order them here without the length limitation of the text box of the group policy.
References
[1] Hardening SSL Cipher Strength and SSL Protocol Support on ISA Servers
http://blog.msfirewall.org.uk/2008/10/hardening-ssl-cipher-strength-and-ssl.html
[2] System Requirements
http://www.microsoft.com/forefront/threat-management-gateway/en/us/system-requirements.aspx
[3] Cipher Suites in Schannel (Windows 7)
http://msdn.microsoft.com/en-us/library/aa374757(VS.85).aspx
[4] Schannel Cipher Suites in Windows Vista
http://msdn.microsoft.com/en-us/library/ff468651(v=VS.85).aspx
[5] TLS/SSL Cryptographic Enhancements
http://technet.microsoft.com/en-us/library/cc766285(WS.10).aspx
[6] RFC 6176 - what’s in for you
http://www.carbonwind.net/blog/post/RFC-6176-what’s-in-for-you.aspx
[7] A quick look over some browsers and their SSL/TLS implementations
http://www.carbonwind.net/blog/post/A-quick-look-over-some-browsers-and-their-SSLTLS-implementations.aspx
[8] Internet SSL Survey 2010
http://blog.ivanristic.com/Qualys_SSL_Labs-State_of_SSL_2010-v1.6.pdf
[9] A brief look at the SSL/TLS settings(behavior) of the Outbound HTTPS Inspection on Forefront TMG 2010 RC
http://www.carbonwind.net/blog/post/A-brief-look-at-the-SSLTLS-settings(behavior)-of-the-Outbound-HTTPS-Inspection-on-Forefront-TMG-2010-RC.aspx
[10] A quick one - Setting the preferred TLS Cipher Suite on TMG Forefront Beta 3: adding a little bit of ephemerality
http://www.carbonwind.net/blog/post/A-quick-one-Setting-the-preferred-TLS-Cipher-Suite-on-TMG-Forefront-Beta-3-adding-a-little-bit-of-ephemerality.aspx
[11] Prioritizing Schannel Cipher Suites
http://msdn.microsoft.com/en-us/library/bb870930(v=VS.85).aspx
[12] How to Restrict the Use of Certain Cryptographic Algorithms and Protocols in Schannel.dll
http://support.microsoft.com/kb/245030
[13] Using Mail Protection with Exchange EdgeSync on Forefront TMG
http://technet.microsoft.com/en-us/library/ee513174.aspx