Firefox may return a ssl_error_rx_record_too_long error when trying to access certain secure web sites through Forefront TMG 2010 RTM doing Outbound HTTPS Inspection

When trying to access certain web sites through Forefront TMG 2010 RTM doing Outbound HTTPS Inspection, Firefox 3.5.x threw this error:


Same web site accessed on the same machine with IE8 or Google Chrome was fine.

According to this, the error code displayed by Firefox means:
This generally indicates that the remote peer system has a flawed implementation of SSL, and is violating the SSL specification.

Instead of inflating this message, let’s analyze the situation a little bit.

Upon a closer look, turns out that Firefox was a SecureNAT client, while IE8 and Chrome were web proxy clients(they share the same proxy settings) + SecureNAT clients(the expression is not the most fortunate, simply put, on the same machine which had TMG as its DG, Firefox was not configured to use TMG as its web proxy, while IE8 and Chrome were), the web site used a wild card certificate, and as an extra note a reverse DNS lookup for the IP address of this web site failed.


This was logged on TMG:


So what actually happened ?
Forefront TMG 2010 RTM’s Outbound HTTPS Inspection does a name check over the server’s certificate. This is similar to what browsers do, if there is a mismatch between the name you’ve typed in the browser’s address bar and the name from the server’s certificate, the browser will warn you about this.
Forefront TMG 2010 does the same thing, in a certain way. This is an extra security check. Please note that Outbound HTTPS Inspection solutions from other vendors may not do a name check over server’s name from the server’s certificate.

The difference between Forefront TMG 2010 RTM and the browser is that Forefront TMG 2010 RTM in certain situations will not have any idea what server the user wants to reach, as unlike the browser, it will not see what the user has typed in the browser’s address bar.

When a web site will be accessed by a web proxy client(like IE8 or Chrome above), Forefront TMG 2010 RTM might have a chance to see what server the user wants to reach, as the client will issue a CONNECT request for the needed address:


In the case of the above web site, while the server has a wild card certificate, the name requested by the client should “match” though. This is probably why IE8 or Chrome could access that site.


With Firefox being a SecureNAT client, the situation might be a little different.
Firefox will open a regular TCP session to port 443 of the IP address of the needed server, and probably issue a Client Hello message. But it will not speak directly with the needed server due to the Outbound HTTPS Inspection. Note that actually Firefox does use the server_name TLS extension telling to the server the server’s name it wants, but I’m not sure this is currently relevant for Forefront TMG 2010 RTM and its Outbound HTTPS Inspection.


Forefront TMG 2010 RTM will connect to that IP address and will get the server’s certificate. At this stage, TMG probably does not have a clue about the server’s name requested by the client as it only saw the IP address of this server(if it doesn’t check the server_name TLS extension). So it needs to find the server’s name somehow if it wants to do a name check.
It can do a reverse DNS lookup, which in this case fails. Note that such lookups might return ambiguous results, for example, going a little bit off-topic from the original case, bellow we have the same error in a similar case(ignore that the server’s certificate is expired as (apparently) bellow TMG does not reach that verification point before denying the connection).


Coming back to our original case, the problem is the wild card certificate and Firefox being a SecureNAT client.
Forefront TMG 2010 RTM can’t take the name from server’s certificate and perform a DNS lookup to see if it will resolve to the same IP address to which the client is trying to connect.
This is what usually Forefront TMG 2010 RTM might do for a SecureNAT client. This may be the reason why other sites like Yahoo mail used to work with this machine and Firefox.
To exemplify for Yahoo mail, I will use the openssl s_client utility on a SecureNAT machine behind Forefront TMG 2010 RTM doing Outbound HTTPS Inspection.
I will first do on an external client a name lookup for, which will return me an IP address(just wait a second to see why I did that):


When I try to connect to this address behind Forefront TMG 2010 RTM doing Outbound HTTPS Inspection, I can’t:


Forefront TMG 2010 RTM connects to this server, gets its certificate(no SAN entries on it), and does a DNS lookup(queries the internal DNS server which is configured to use a forwarder) for the CN obtained from the server’s certificate: As answers for this query it will get some different IP addresses than the one obtained by the external client(which used a different DNS server).
And our connections fails with a name mismatch on TMG(the DNS answers did not say that a server with such a name can be found on the IP address requested by the client).


Instead, if we attempt to connect to one of the IP addresses returned in the above DNS answers, we can successfully do so(I’ve added bellow the corresponding reverse DNS lookup made by TMG for this attempt):


So, for the case of the site for which Firefox returned the ssl_error_rx_record_too_long error:
- as the certificate was issued to ‘*’(wild card certificate), TMG cannot really determine like above for the name of the server to which the client wants to connect.

What’s up with the error displayed by Firefox ?
This is triggered by the unusual response used by TMG(to politely say that this does not appear in TLS 1.0 RFC, actually the RFC says: After sending the client hello message, the client waits for a server hello message. Any other handshake message returned by the server except for a hello request is treated as a fatal error.).
Firefox attempted to initiate a TLS handshake, sent a Client Hello message, message to which TMG responded with a HTTP 1.1 500 Internal Server Error (!), we can see this with Wireshark(which also labels the corresponding packet as ‘Unknown (TLS) Record’), note that Firefox sends an unexpected_message, which according to TLS 1.0 RFC means: An inappropriate message was received. This alert is always fatal and should never be observed in communication between proper implementations.:


Can’t say for sure, but apparently this is expected behavior from TMG (!):
A number of problems can occur with server certificates that result in Forefront TMG blocking access to the site. This action is recorded in the Forefront TMG log while Forefront TMG sends an HTML error page to the client (only a web proxy client will display the error page).

Usually if a server would want to return a HTTP error message over SSL/TLS for HTTPS, it should first complete the SSL/TLS handshake with the client, and send it over the established SSL/TLS connection as a result to a HTTP request received from the client(like in the case of servers “politely” saying that SSL 2.0 is not supported).
Firefox expected a Server Hello, or just a TCP RST segment, not a HTTP 1.1 500 Internal Server Error.
The unusual response from TMG triggered a “similar” answer from Firefox.

For a web proxy client it’s easier for TMG to return a HTTP proxy error for a CONNECT request, which the browser should correctly display(although in practice this would not happen with all the browsers):


Solution used
Configured Firefox as a web proxy client by telling it to use TMG as its proxy server.
After this step, Firefox was able to access this web site through Forefront TMG 2010 RTM doing Outbound HTTPS Inspection.


Comments (1) -

  • Sam
    Thks, this solved my problem. =)
Comments are closed