Forefront TMG
ISA Server
Vyatta OFR

 Updated 21.08.2007
OpenVPN and ISA 2006 - Part 2: OpenVPN on ISA

 - 1. Overview
 - 2. Configure ISA and Configure OpenVPN on ISA
 - 3. Configure OpenVPN on the Client
 - 4. Test the Configuration
 - 5. Issues

 1. Overview
Before we proceed, I want you to remember that OpenVPN is not a third-party software for ISA 2006, so installing it on ISA is not something supported or recommended. Be aware of the fact that you are doing it at your own expense(both security and functionality issues).

In a production environment, you must install OpenVPN on a server behind ISA.

 2. Configure ISA and Configure OpenVPN on ISA
For OpenVPN we will use once again "dev tun", which will create a routed IP tunnel.
So let’s install OpenVPN with GUI on ISA.
Same story with CA. Just follow the same steps.
Now the ISA world:
Create a new Network called OpenVPN and add the bellow ranges to it(obviuosly is not anymore on the DMZ Network, ISA fresh install):

The OpenVPN Network
Figure52: The OpenVPN Network

Create a network rule with a “Route” relationship between OpenVPN and Internal Networks:

The ‘Route” Network Relationship between OpenVPN and Internal
Figure53: The ‘Route” Network Relationship between OpenVPN and Internal

And one with a “NAT” relationship between OpenVPN and External Networks:

The “NAT” Network Relationship between OpenVPN and External
Figure54: The “NAT” Network Relationship between OpenVPN and External

The ISA Networks discussion: looking at the TAP adapter, ISA will see that it is directly connected to network This one is part from the OpenVPN network. But the OpenVPN network includes So we need to add some routes for the “rest”:

Adding routes on ISA
Figure55: Adding routes on ISA

So ISA’s routing table will look like:

ISA’s routing table
Figure56: ISA’s routing table

This time the Server Publishing Rule will be for instead of

The HTTPS Server Publishing Rule
Figure57: The HTTPS Server Publishing Rule

What the TAP adapter will try to do is to get an IP address from the DHCP server. It will fail because ISA it will block it.
We need the bellow rules on ISA:

The DHCP Rules
Figure58: The DHCP Rules

Now go and start the OpenVPN service(set it to automatic to avoid errors when restarting ISA).

The OpenVPN Service
Figure59: The OpenVPN Service

The TAP adapter should be able to obtain a “long” lease:

The TAP Adapter on ISA
Figure60: The TAP Adapter on ISA

Obvioulsy there will be alerts on ISA about network definitions. Simply restart the machine. If everything is configured properly, they will not appear anymore. If not, debug it.

Also you might get a warning in the OpenVPN log that it failed to add a route to MASK through But it is OK since we already have taken care about ISA's routing table just to make it happy.
After the machine restart you can change the bellow rule replacing External with OpenVPN:

Modifying the DCHP rule
Figure61: Modifying the DCHP rule

The server config(the IP address to which OpenVPN listen is
#remote myremote   # server mode
port 443  #port used by OpenVPN
proto tcp-server
dev tun
server   # the subnet used by clients
push "redirect-gateway"   # no split tunneling
push "dhcp-option DNS"
push "dhcp-option WINS"
push "dhcp-option DOMAIN"  # some DHCP options
#the locations of the keys
dh "C:\\Program Files\\OpenVPN\\easy-rsa\\keys\\dh2048.pem"
ca "C:\\Program Files\\OpenVPN\\easy-rsa\\keys\\ca.crt"
cert "C:\\Program Files\\OpenVPN\\easy-rsa\\keys\\server.crt"
key "C:\\Program Files\\OpenVPN\\easy-rsa\\keys\\server.key"  # This file should be kept secret
# the tls-auth directive adds an additional HMAC signature to all SSL/TLS #handshake packets for #integrity verification
tls-auth "C:\\Program Files\\OpenVPN\\easy-rsa\\keys\\ta.key" 0 
cipher AES-128-CBC   # AES you can use AES-256-CBC if you want
# enable LZO compression
# moderate verbosity
verb 4
mute 10

Again a test rule allowing all from OpenVPN to Internal and vice-versa:

Allow All between Internal and OpenVPN
Figure62: Allow All between Internal and OpenVPN

And a rule for Internet access for OpenVPN clients:

Internet Access for VPN clients
Figure63: Internet Access for VPN clients

 3. Configure OpenVPN on the Client
The client configuration remains the same like in part 1.

 4. Test the Configuration
A quick connectivity test:

A quick connectivity test
Figure64: A quick connectivity test

A ping test(good for name resolution too):

Testing: ping and name resolution
Figure65: Testing: ping and name resolution

Don’t bother to ping because it’s firewalled.
We can enabled the Web Proxy on the OpenVPN ISA Network and use it in our browser:

Optional: the Web Proxy
Figure66: Optional: the Web Proxy

The ISA’s logs:

ISA’s logs
Figure67: ISA’s logs

If you have reached this point then by now you should have a functional SSL VPN server on your network.

 5. Issues
Yes, there are some issues.

Now imagine that we have an Exchange Server, say 2003. ISA, a domain member.

A typical scenario will be to publish OWA with SSL bridging, FBA on ISA(which first validates credentials using Windows(Active Directory)) and with authentication delegation set to basic authentication(after ISA validates credentials will forward them as basic credentials to the Exchange server). Our rule has configured ISA to listen for requests from the External Network(single IP address on the External interface) and from the Internal Network(again single IP address on the Internal interface), TCP ports 80(for HTTP to HTTPS redirection) and 443 used on both interfaces. So no packets are actually reaching the Exchange server until ISA validates credentials. SSL all the way(between client and ISA and between ISA and Exchange server).

What will happen with the above configuration(OpenVPN on ISA and OWA published) is very simple and logical: some alerts(Server Publishing Failure) for the OpenVPN and OWA publishing rules. ISA is unable to bind a socket for the server. IP addresses involved: and the OpenVPN server config file, is listening on this IP address), so the rules will fail. For example ISA has a socket for OWA and therefore cannot create another one for OpenVPN So OWA will work from External but OpenVPN not. If OpenVPN is able to "get" the socket then OWA will fail from Internal.

Server Publishing Failure
Figure68: Server Publishing Failure

A very simple and logical solution is to have two different IP addresses on ISA's External interface(you need two public IP addresses) and one more NIC in order not to add another IP address to the Internal interface of ISA(our DMZ NIC from the first part, since it is unused now, will do the trick).

Publish OWA with one external IP address and OpenVPN with the other external one. Configure the OpenVPN to listen on and not on the publishing rule accordingly too) since "OWA will listen" on So the sockets will look like: 192.168.234:443 and for OWA, 192.168.235:443 and for OpenVPN.

The rest will remain unchanged, obvioulsy except the fact that the OpenVPN clients will use the new external IP address.

In OpenVPN's 2.1 Man Page there is a directive "--port-share" which enables OpenVPN to: "When run in TCP server mode, share the OpenVPN port with another application, such as an HTTPS server." But there it says that it is not implemented on Windows. Also the "application" involved here is the ISA firewall....

So unless you have two public IP addresses on your ISA you will have a problem. I did not spent too much time testing this issue. Anyway, OpenVPN is not a third-party add-on for ISA, so in my opinion it is better to place it on an ISA's DMZ network(you still need two public IP addresses if you want OWA and OpenVPN to use same TCP port 443 on ISA's External interface in order for ISA to bind different sockets for them) and from there traffic will flow through ISA(therefore will be filtered by ISA) to other ISA Networks.

I was "pushing" on using TCP port 443 in a hope for better connectivity from everywhere. As we already saw it is better to use UDP and so some problems will dissapear. OpenVPN has its own official IANA UDP/TCP port: 1194. So if do not experience such connectivity problems go ahead and use UDP port 1194.

On the the client just comment the following lines like below:

#port 443

#proto tcp-client

On the server do the same for:

#port 443

#proto tcp-server

On ISA modify the server publishing rule by creating a customs protocol called OpenVPN for UDP port 1194:

ISA Custom OpenVPN Protocol
Figure69: ISA custom OpenVPN protocol

After this modification our failed server publishing rule comes up:

Rule Recovered
Figure70: Server Publishing Recovery

If you want for some reasons to use a custome TCP port say 5000, just create on ISA a server publishing rule using "HTTPS Server"and click the "Ports" button and specify your port.

Custom HTTPS ports
Figure71: Using custom HTTPS ports for the HTTPS server publishing rule

For more information about the multitude of options that OpenVPN offers please reffer to its documentation:

I have just use few of OpenVPN's options.

It was only a brief view.
In the future maybe we will take a look at using bridging instead of routing in the relationship of ISA-OpenVPN.