ISA Server 2006 as a VPN server and the selection of the certificate to be used for IKE authentication for L2TP/IPsec connections – a possible work around

- ISA Server 2006, domain member, its Computer Certificate Store, Personal Store contains some certificates, say a few from the Windows enterprise CA, others from various public CAs.
- L2TP/IPsec connections, IKE authentication with certificates from the Windows enterprise CA.
- not sure which certificate from the enterprise CA will ISA(actually not ISA itself, rather the Windows OS) use for IKE authentication with the VPN clients.
- if a “VPN client” attempts to use for IKE authentication a certificate(usable for IKE authentication) from one of the other public CAs, ISA will use for IKE authentication the certificate(if usable for IKE authentication) it has from this CA(due to the default Windows L2TP/IPsec policy).

What if:
- ISA will select the certificate you want from the Windows enterprise CA for IKE authentication.
- even if the client uses for IKE authentication a certificate(usable for IKE authentication) issued by one of the other CAs, ISA will still accept for IKE authentication only a certificate from your enterprise CA. For this see the bellow note.

Interesting, eh ?

Note: What it is described bellow, it’s not a certificate restriction, just makes ISA to select every time for IKE authentication the certificate from the enterprise CA you want, if you want ISA to accept certificates issued only by your enterprise CA from the VPN clients, you need to define a custom IPSec policy for L2TP, maybe as described here(a more delicate job).

When having multiple “computer” certificates on ISA within the Computer Certificate Store, Personal Store from the same enterprise CA, a problem may appear with Vista or Windows 7 RC L2TP/IPsec VPN clients, that is, these clients can be configured to verify the VPN server’s certificate(highly recommended) for IKE authentication(machine authentication) and ISA(the Windows OS) cannot be configured to select a specific certificate for IKE authentication(so even if you guess which certificate ISA uses now, maybe tomorrow when you add a new certificate from the enterprise CA say to publish a new secure web site(and you don’t want to use a public CA for the SSL certificate), ISA will not pick the same certificate for IKE authentication…).

The option that enables the extra checks on the VPN server’s certificate is called Verify the Name and Usage attributes of the server's certificate, see this and this for more details about it. Before you hit me with the access is controlled per user, remember that without the L2TP/IPsec VPN client verifying the VPN server’s identity, a user with restricted privileges per firewall policies may be able to “elevate” his or her privileges. –;)

What it’s checked:
- the server’s certificate must have the EKU field containing Server authentication(
- the name that appears on the server’s certificate(say within the CN field) must be the same with the name of the VPN server you connect to(configured on the VPN connection). If on the server’s certificate, both the CN and the Subject Alternative Name(set to DNS Name) appear on the server’s certificate, then the SAN field must match the name of the VPN server you connect to. If you type an IP address as the address of the VPN server on your VPN connection, and the certificate contains a “host name”, the check over server’s name will not take place(I don’t know what it will happen if the server’s certificate contains instead of a name an IP address, I did not test it, anyway likely very few people use such certificates).

I was messing with IKEv2 in Windows 7 RC and with the certificates possibilities for the VPN server, trying to see what will keep happy both IKEv2 and L2TP/IPsec. As I’ve already mentioned here, apparently if you use on ISA a certificate with the EKU field containing IP security IKE intermediate (, this certificate will be preferred for IKE authentication by the Windows API. But Vista L2TP/IPsec VPN clients with the extra checks enabled don’t like such an “only” EKU field one from the server, they, like IKEv2 clients in Windows 7 RC, want the EKU field to contain Server authentication(

So what if we create a certificate containing within the EKU field both the Server authentication and IP security IKE intermediate ?

Well, it looks like we have a work around to make ISA(the Windows API) select the certificate from the Enterprise CA we want to and keep the Vista and Windows 7 RC L2TP/IPsec VPN clients happy(the XP clients are pretty happy by default –:) ).
I’m not a Mac user, so currently I do not have a Mac to test with such a certificate, but I think that it might work with the built-in Mac L2TP/IPsec VPN clients too, as the certificate contains the IP security IKE intermediate EKU. If so, it would be great, as it solves both the Vista and Mac L2TP/IPsec VPN clients issues.

Note that in the bellow lines the ISA machine was a domain member.

So how are we going to get such a certificate from the enterprise CA ?
From an OpenSSL CA this would be easy, but it is likely that a Windows enterprise CA to be used.

Pretty simple, we are going to craft our very own custom computer certificate template. Note that for that it looks you need the enterprise CA installed on Windows Enterprise Edition or so, at least for Windows 2003(I could not see my custom certificate template when I want to add it as a Certificate Template to Issue), maybe this article and this article will provide your more details.


On the enterprise CA, open the Certification Authority mmc, right-click the Certificate Templates and click Manage:


This will open the Certificate Template window. Right-click the IPSec certificate template and click Duplicate Template:


The Properties of New Template window appears, on the General tab, I’ve named this template My IPsec:


On the Request Handling tab of the Properties of New Template window, I’ve checked the Allow private key to be exported option(if you use ISA EE and NLB you need to install the same certificate on every array member, so you will want to be able to export the certificate) and set the Minimum key size to 2048. Actually the screens from bellow with the VPN connections were taken with two ISA Server 2006 EE using NLB unicast mode, acting as VPN servers, same custom certificate installed on both ISAs.


On the Subject Name tab of the Properties of New Template window, I’ve chosen for Subject name format the Common Name and leaved in the Include this information in alternate subject name the DNS name:


I did not touch the Issuance Requirements tab or the Superseded Templates tab of the Properties of New Template window:



On the Extensions tab of the Properties of New Template window, select Application Policies and click the Edit button:


On the Edit Application Policies Extension click the Add button:


On the Add Application Policy window select Server Authentication and click the OK button:


And now Server Authentication appears on the Edit Application Policies Extension window. Click the OK button on this window:


And we’re done with the Extensions tab of the Properties of New Template window:


The Security tab of the Properties of New Template window is fine by default, the domain users(they are not added within the Group or user names area by default) cannot enroll certificates using this template and domain computers can enroll a certificate using this template(this template cannot be used from the Web enrollment):



Click OK to close the Properties of New Template window.

Now we can see our custom template within the Certificate Template window:


Close the Certificate Template window.

On the Certification Authority mmc, right-click the Certificate Templates, click New, and click Certificate Template to Issue:


On the Enable Certificate Templates window select the custom template we’ve just created and click the OK button:


And now on the Certification Authority mmc, within the Certificate Templates, we can see our custom template:


So now we are ready to start issue such certificates.


We are going to use the Certificates mmc, to request such a certificate to be installed in the Local Computer Certificates Store.

We can request the certificate from the ISA machine, or use another domain member machine(obviously you should not do that from a regular PC), then export that certificate(and delete it from this machine) and import it on ISA. Note that the name of that machine will appear on the issued certificate. On ISA, the request may fail, you need to uncheck the Enforce strict RPC compliance setting of the Active Directory System Policy, and to create a temporary(till you request the certificate) allow all access rule from localhost to the Internal Network(the location of the CA).

For example, in the bellow lines I will request such a certificate from a different machine(a Windows 2003), just to see if the custom certificate will be selected by ISA when multiple “computer” certificates are within Computer Certificate Store, Personal Store from the same enterprise CA like a “normal” computer certificate from the enterprise CA(using the Computer Template) or a web server certificate from the enterprise CA.


Click Start, click Run, type mmc and click OK.

Click the File menu, from the File menu click Add/Remove Snap-in, click the Add button on the the Add/Remove Snap-in window, Standalone tab. Add the Certificates standalone snap-in:



Select the Computer account and click Next.


Select the Local Computer and click Finish.


Expand the Certificates (Local Computer) node, right-click the Personal, click All Tasks and click Request New Certificate:


The Welcome to the Certificate Wizard Request appears, click Next on it.

Select the custom certificate type we just created and click Next(we specified the minimum key size so we don’t need to click the Advanced option):


Type a friendly name, something to help you easily identify this certificate and click Next:


And click Finish:


Click OK on the The certificate request was successful window:


Let’s take a quick look at the certificate, we have a CN, we have the needed SAN entry and the EKU fields, the Key Usage is fine:






Now I’m going to export this certificate along with its corresponding private key, delete it from the “dummy” machine and import it on ISA within the Computer Certificate Store, Personal Store. And reboot ISA. I will not picture/describe these steps, as it is already explained over the web how to export a certificate and import it on another Windows machine. For example see this doc.


A quick look at the the Computer Certificate Store, Personal Store on ISA, and we can spot our custom certificate, along with other two from the same enterprise CA, a “normal” computer certificate issued to ISA(which has no purpose right now) and a web server certificate used to publish OWA. Let’s see if the needed one from these three will be selected for IKE authentication.



I’m going to connect from a Vista Ultimate SP2 RC machine(the VPN client must be able to resolve the VPN server’s name, a quick work around, until the needed public DNS entry is configured, during tests, is to use the hosts file on the VPN client's machine):



And we are connected:



Let’s take a quick look at the Oakley.log on ISA. As can be seen from bellow, ISA accepts IKE authentication with certificates from these two CAs, one is our enterprise CA, the other one is the one that issued the certificate for a secure web site that is published with ISA(this certificate can be used for IKE authentication), remember that we are using the default Windows IPsec policy for L2TP:


So ISA, as per using the default Windows IPsec policy for L2TP, will request for IKE authentication a certificate from one of the two CAs:


The VPN client uses a user certificate from the enterprise CA stored within its Computer Certificate Store, here it’s its response:


Next ISA will select its certificate to use for IKE authentication, the certificate we want, and as can be see from bellow, from ISA’s point of view IKE MM was established:


And the client accepts ISA’s credentials and starts IKE QM negotiations:



I’m going to add one more certificate on ISA from the enterprise CA, another web site certificate(the one with abc), to see if this will change the outcome:


And no change when the same client connects, ISA selects the certificate we want, pretty nice:



What if I delete our IPsec certificate, which certificate will be chosen ?


So, prepare the bets and, it is the owa one that was selected for IKE authentication:


Obviously the Vista L2TP/IPsec VPN client is not happy about this, due to the invalid name on the certificate, and will not accept ISA’s IKE credentials:



Let’s delete the owa certificate too:


Prepare the bets one more time and now it is the abc one, which is pretty funny, as the “normal” computer certificate(using the Computer certificate template) issued to ISA is pretty much useless:




As said before, this is not a certificate restriction, if a “VPN client” uses a certificate from the third-party CA(maybe a stolen certificate or so), ISA will use its certificate from that CA, and there is a chance that IKE authentication to be successful(if the certificate used by the “VPN client” is still valid):




If a custom IPsec policy for L2TP would have been in place instead of the default Windows IPsec policy for L2TP, the above IKE credentials from the client would have been rejected by ISA:





All of the above screens for teh VPN connection were taken with Window Vista Ultimate x64 SP2 RC, Windows 7 RC x64 and Windows XP SP2 as L2TP/IPsec VPN clients and ISA Server 2006 SP1 EE using NLB unicast mode as the VPN servers(bellow there are two VPN clients connected, one using Window Vista Ultimate x64 SP2 RC and the other one Windows 7 RC x64 ):


Comments (3) -

  • Adrian,

    * Stop the ISA Control Service
    * Open RRAS console
    * Go to Authentication and select the required certificate from the drop down list for IPSec (the same as defined for validation in the CMAK profile)
    * Save RRAS changes
    * Start ISA Services

    This process always seems to work for me; have I just been fooling myself, or did I get lucky all this time? ;)


    • Hi Jason,

      I'm not sure I'm following you. Smile

      Where is on the RRAS "the drop down list for IPSec" ?
      Can you please detail on that ?

      Perhaps are you referring to the EAP-TLS user authentication ?
      This indeed is not a problem, if you use EAP-TLS, and indeed you can select like so the certificate you want for EAP-TLS.

      But I'm talking about machine authentication.
      As can be seen from the above Oakley.logs, IKE searches for an "IPsec usage" certificate, so if you use the "needed" EKU on that certificate, this certificate will be selected. But if you don't have the server authentication EKU on this certificate too, Vista and Windows 7 RC L2TP/IPsec VPN clients with the extra checks enabled, will not accept the VPN server's credentials and machine authentication will fail.
      If you skip this validation on the L2TP/IPsec VPN client, in certain circumstances, the EAP-TLS validation done by the L2TP/IPsec client during the user authentication phase may value as low as .02 cents...

  • Ah, ok my bad! Yes, I was talking about the EAP…



Pingbacks and trackbacks (1)+

Comments are closed