When you are troubleshooting L2TP/IPsec remote access VPN connections using certificates for IKE authentication, a good place to start, you may think, would be the Event Viewer on your ISA Server, particularly the Security Audits.
I would add: maybe. It might be a good idea or not. Depends.
How is that, you may wonder ?
For example you may see that Event ID 541: IKE security association established, Mode: Key Exchange (Main mode) was logged.
So, you would say: IKE MM negotiations went fine, I have an established IKE MM Phase, and now IKE Quick Mode may start.
Wrong. You have an IKE MM Phase established just from ISA's point of view.
That does not imply that the client has also an established IKE MM Phase.
The client will send to ISA its third IKE MM message, an encrypted message, which contains the authentication payload along with the client's certificate.
ISA analyzes this message, everything appears to be fine, and it logs that IKE MM was establised.
After that ISA sends to the client its third IKE MM message, which contains the authentication payload along with ISA's certificate.
The client will take a look at this message. If the client(say it's a Mac OS X L2TP/IPsec VPN client or a Vista L2TP/IPsec VPN client) is configured to perform additional checks on ISA's certificate, it may not accept ISA's certificate, for example the SAN field on the VPN server's certificate does not contain the expected DNS name.
Let's exemplify the above described issue.
I recently wrote something about Mac OS X L2TP/IPsec as VPN clients and ISA as an L2TP/IPsec VPN server:
http://www.carbonwind.net/ISA/MacOSXVPNL2TP/MacOSXVPNL2TP1.htm
There we have seen that the Mac OS X L2TP/IPsec VPN clients perform certain checks on the VPN server's certificate. So a Mac OS X L2TP/IPsec VPN client can serve as a good example for a quick test.
Let's say I'm connecting from a Mac OS X L2TP/IPsec VPN client to an ISA 2006 L2TP/IPsec VPN server, and the VPN connection cannot be established. ISA has installed on it a certificate which contains within the SAN field the DNS Name isabranch.carbonwind.net.
The Mac OS X L2TP/IPsec VPN client is configured to connect to the vpn.carbonwind.net VPN server.
The Event Viewer/Security on ISA will log an Event ID 541: IKE security association established, Mode: Key Exchange (Main mode).
If we look at the Oakley.log on ISA, we can notice that indeed from ISA's point of view the VPN client was successfully authenticated and the IKE MM phase was established. And that ISA is using a certificate with the CN isabranch.carbonwind.net. If we look into the Computer Certificate Store on ISA, we will see that this certificate has within the SAN field the DNS Name isabranch.carbonwind.net.
ISA will send its third IKE MM message to the VPN client.
However, the Mac OS X L2TP/IPsec VPN client sends back a NOTIFY message, informing ISA that its certificate is invalid(ISA does not appear to be aware of the meaning of this message).
The Mac OS X L2TP/IPsec VPN client expected the SAN field from ISA's certificate to contain the DNS Name vpn.carbonwind.net.
So from the Mac OS X L2TP/IPsec VPN client's point of view, IKE MM was not quite established, the authentication of the VPN server failed.
If from the VPN client's point of view the VPN server would be successfully authenticated and the IKE MM would be established, the VPN client may send the first IKE QM message. But in our case, the Mac OS X L2TP/IPsec VPN client sends back an ISAKMP Informational Exchange message instead, so no IKE QM negotiations. The second authentication phase, user authentication(PPP authentication), will start only after IKE QM negotiations will be successfully completed, the L2TP tunnel started and a PPP authentication method selected.
Thus, it must be noted that the Event Viewer is just a quick troubleshooting tool for IKE negotiations, and it may be a little bit confusing. If you want to analyze deeper the IKE negotiations, check the Oakley.log on ISA.