This is an exciting week for our team! At TechEd North America we provided a sneak preview of new cloud functionality which complements the Web Application Proxy role that is part of Windows Server 2012 R2. This is another step in assuring that we provide our customers with the best solutions to make their Web applications available to remote users. Microsoft is committed to remote solutions, and we plan to continue to evolve both of them.
This is a good opportunity for us to start to deepen our discussions with you, the community that deploys and maintains our solutions. We will use this blog to share less formal information than can be found in our TechNet documentation. We plan to provide lots of tips, tricks and explanations that we want you to be aware  of to help you with your deployments. Later this week we will post here link to the TechEd session and some interesting news about a Web Application Proxy update.
In this blog you will see posts from the product group, the support team and from Microsoft consultants. We are spread across three continents and too many time zones. Most of us are long time veterans in this field who started out working on  legacy application proxy solutions that were part of TMG, UAG, ISA and IAG.
We look forward to hearing from you as well and welcome any feedback and suggestions on content you would like to see.

Welcome aboard!

Hi,

The recorded Web Application Proxy session in TechEd North America 2014 is now live on MSDN channel 9 Web site: https://channel9.msdn.com/Events/TechEd/NorthAmerica/2014/PCIT-B327

The cloud service part begins at 00:51:35.

Enjoy...

Now that TechEd North America 2014 is over and I'm sure you have finished watching the Web Application Proxy introduction session you are probably looking for more.  Here are other sessions available on-line with Web Application Proxy content:

1. Troubleshooting Active Directory Federation Services (AD FS) and the Web Application Proxy- great session by John Craddock on troubleshooting. He is taking you to a deep dive into the system internals. Please note that the Web Application Proxy misconfiguration that he shows at the end is not relevant after the May update as this value updates automatically - we will have a blog post on that soon.
   
   
2. Publishing Microsoft Exchange Server: Which TLA Should You Choose? - Greg Taylor reviews all the options of publishing Exchange and explains why [spoiler alert!] Web Application Proxy shall be used if you don't want to put your Exchange servers directly on the Internet.
   
   
3. Active Directory + BYOD = Peace of Mind - Mahesh Unnikrishnan explains the entire BYOD solution across ADFS and Web Application Proxy.
   
   
4. Understanding Microsoft’s BYOD Strategy and an Introduction to New Capabilities in Windows Server 2012 R2 - Adam Halls explains the overall strategy and how Web Application Proxy enables multiple scenarios.

I would use this opportunity also to recommend Jesse Esquivel's blog post on publishing Exchange 2010 OWA with smartcard-only login (no usernames or passwords) using Kerberos Constrained Delegation (KCD).

Enjoy,

Meir Mendelovich, Senior Program Manager, Application proxies product group

Firstly, thanks to Meir for his opening intro blogs.

As our first deep technical blog we thought we’d throw you in at the deep-end and the below article intentionally has quite a lot of technical depth to help give you some insight into some of the architectural workings of Web Application Proxy.

Over the coming months we’ll be bringing a good mix of content to you at differing technical levels but this one is definitely towards the deeper end of the technical spectrum. We’ll look forward to hearing if this is too much detail or if you’d like to see more of the same.

Introduction

As people are rolling out Web Application Proxy deployments we’ve seen some limited issues where the Proxy Trust relationship between Web Application Proxy and AD FS 2012 R2 is unable to establish correctly or subsequently fails often due to issues with the Certificate Trust List being sent by the AD FS server. We wanted to take the time to share our findings and explain some of the underlying architectural principles involved.

Issue Symptoms

The symptoms that we have been seeing are that the Proxy Trust relationship between a Web Application Proxy server and an ADFS 2012 R2 server either cannot be correctly established or starts to fail at some point in time. The symptoms of this can be found in the AD FS event logs on both the Web Application Proxy server and AD FS 2012 R2 server.

In the AD FS event log on the Web Application Proxy server:- 

In the AD FS event logs on the AD FS server:-

These events show that the Web Application Proxy was presenting a client authentication certificate when trying to retrieve the proxy configuration data but that the AD FS server is not seeing the certificate for some reason.

Background Information – how does the Proxy Trust work?

The proxy trust relationship between a Web Application Proxy server and the AD FS 2012 R2 server is client certificate based. When the Web Application Proxy post-install wizard is run a self-signed Client Certificate is generated and inserted into the AD FS configuration store using the credentials specified in the wizard. AD FS also propagates this to the AdfsTrustedDevices certificate store on the AD FS server.

The Proxy Trust certificate is then used by the Web Application Proxy server to authenticate to the AD FS server. This also means that the Proxy Trust is independent of domain membership and that the Web Application Proxy does not need to be domain joined. The Common Name on the certificate is “ADFS Proxy Trust – machinename”

When the client certificate authentication is negotiated, the AD FS server sends a Certificate Trust List (CTL) based on the contents of the AdfsTrustedDevices store. A CTL allows a client to filter valid client certificates from all the client certificates that may be present. If the CTL list of filtered certificates returns only a single certificate which is presented without the user needing to manually select a certificate.

If Device Registration has been enabled the AdfsTrustedDevices store also contains the MS-Organization-Access certificate which is the certificate used to issue Workplace Join client certificates.

This is important from an AD FS perspective primarily from a Workplace Join certificate as the CTL includes filtering for Workplace Join certificates so that a client doesn’t get unexpected and potentially confusing client certificate authentication prompts when accessing Workplace Join secured resources.

How Does this Relate to the Problem?

In a number of cases, we have seen that the CTL list sent by AD FS to the Web Application Proxy server was incorrect.

This was causing the Web Application Proxy server Proxy Trust cert to be filtered out as a valid certificate leading to no client certificate being presented to AD FS and the Proxy Trust failing with the event logs errors detailed above.

Where Does the AD FS Certificate Trust List come from?

So, we now know that the issue is due to an incorrect CTL being presented we now need to understand how the CTL is generated.

As many of you will know, AD FS 2012 R2 does not run on IIS as previous versions of AD FS did but instead runs directly as a service on top of http.sys. We need to understand a little bit about how AD FS uses http.sys in order to understand the root cause of the proxy trust issues.

From an http.sys perspective, AD FS uses hostname:port bindings as opposed to IP:port bindings. This is beneficial as we can host multiple disparate SSL certificates on the same IP/port instead of having to have a separate IP address for each certificate we want to bind to port 443. This is mainly leveraged in Web Application Proxy’s web application publishing but the same principles are used in AD FS 2012 R2.

When a client sends an SSL Client Hello it sends a Server Name header which details the FQDN that the client is connecting to. The server can then use this header to decide which certificate to serve. This is known as Server Name Indication.

Note - we will be following this blog up with a subsequent blog on SNI as there are some interesting discussions to be had about how to support clients who don’t send an SNI header.

Let’s take a look at what an AD FS certificate binding looks like. We can do this using the http.sys netsh sslcert command from the command prompt:-

In the above netsh output you will see a number of different certificate bindings, the above excerpt shows the main AD FS port 443 binding. As you can see this is a hostname:port binding using the AD FS service FQDN. The sharp-eyed among you will also have spotted that the Ctl Store Name is set to AdfsTrustedDevices. This is the certificate store that http.sys will use to generate the CTL list and is set by AD FS when it creates the SSL Certificate binding.

If the CTL Store name is not specified then Windows 2012 R2 will fall back to use a one of number of different certificate stores as detailed in the “Management of trusted issuers for client authentication” section of the following article:-

http://technet.microsoft.com/en-us/library/hh831771.aspx

In practice, on an AD FS server this will mean the Trusted Root Certification Authority store is used.

Also, as the above article also mentions, CTL generation from a CTL store can also be affected by having a CA Issued certificate in a store where you would typically only have self-signed certificates such as the Trusted Root CA store as noted in http://support.microsoft.com/kb/2802568. Root Certification Authority certificates are by definition self-signed.

http.sys SSL Certificate bindings and order of precedence

An equally important piece of information we need to understand in relation to the issues we are looking at here is the http.sys order of precedence for SSL Certificate bindings. HTTP.sys uses the following order of precedence for its SSL bindings:-

• Specific IP:port binding

  If you have an IP:port binding for the IP address the inbound traffic is sent to then this binding will be used e.g. 1.2.3.4:443

• Hostname:port

  If there is no specific IP:port binding in place then http.sys will try to match a certificate using the SNI header in the SSL Client Hello and the Hostname:port bindings e.g. adfs.contoso.com:443

• Wildcard IP:port binding

  http.sys also supports a wildcard:port binding for 0.0.0.0:443 which will be used as a final fallback if we have not match on a specific IP or hostname binding.

As we will see later on, this becomes very important and conflicting or incorrect bindings can lead to CTL issues. The above is actually a slightly simplified version of the full http.sys certificate precedence. The full list for Windows Server 2012 R2 is as follows:-

Known Causes of Proxy Trust failures

Keeping of all the above background information in mind let’s look at a number of different root causes we have seen for these Proxy Trust issues over the last few weeks along with resolutions for these.

Root Cause 1 – a specific IP:port certificate binding on the AD FS IP address

As we know, the Certificate Trust List is generated based on the CTL Store Name defined on the http.sys certificate binding. We can use “netsh http show sslcert” output to examine the certificate bindings from a problem AD FS server. In a number of customer cases we’ve found a binding similar to the following:- 

There are 2 important things to note here:-

1. This is an IP:port binding where 1.2.3.4 is the IP address of the ADFS server. This is also the IP address that the ADFS FQDN resolves to.
2. The Ctl Store Name is null so the OS will fallback to use the Trusted Root CA store to build the Certificate Trust List

In this case as there is a specific IP:port binding in place this binding was being used ahead of the ADFS Hostname:port binding. Despite the certificate being used was exactly the same as the IP:port binding does not have a Ctl Store specified the Trusted Root CA store was being used to build the CTL list causing the Proxy Trust certificate on Web Application Proxy to be filtered out.

The only missing piece of the puzzle is why this binding was in place and, in the above example, the Application ID helped us understand this.

                Application ID               : { 4dc3e181-e14b-4a21-b022-59fc669b0914}

This happens to be the SSL Certificate Owner Application ID for w3svc (IIS) so it looked as if the customer had configured this binding in IIS and this proved to be the case. We have seen a number of customers install IIS on the AD FS server either due to believing it is required for AD FS 2012 R2 (it isn't) or as a way to generate a Certificate request.

Note – when you configure http.sys certificate bindings using netsh you can configure whatever Application ID you want so a binding may have been created manually and appear to have been configured by an application.

The PowerShell script provided in the ‘Issue Detection PowerShell Script’ section below can be run on an AD FS 2012 R2 server to determine if this issue is present.

Root Cause 1 - Resolution Options

When looking at a resolution the first thing to determine is why the binding is in place.

This might have been created by mistake or as a leftover from an install of a different product or Windows component but it may equally be there for a valid reason. As mentioned above it’s also possible that a binding may have been manually created possibly to support non-SNI clients.

Once you have determined the reason for the binding being in place, you can decide which of the following resolutions is more appropriate:-

1. Remove the specific IP:port binding

Be sure to check that the binding does not come back as if there is an application configured with such a binding it may recreate this automatically or on next service start-up.

2. Use an additional IP address for AD FS traffic

In our example above if the 1.2.3.4 SSL Binding is expected and required, then using a 2^nd IP such as 1.2.3.5 for ADFS and resolving the ADFS service FQDN to this IP address would mean that the Hostname:port bindings would then be used (assuming that there isn’t an IP:port binding for 1.2.3.5:443 of course).

3. Configure the AdfsTrustedDevices store as the Ctl Store for the specific IP:port binding

This will again have some dependence on why the specific IP:port binding is there and if this relies on the default CTL Store for client certificate authentication but an option would be to set the Ctl Store on the IP:port binding to be the AdfsTrustedDevices store

At this point in time, our recommendation is to avoid having IP specific bindings and use option 1 as the preferred resolution. 

Root Cause 2 – the AD FS Hostname:port binding does not have a Ctl Store defined

This was a bit more of a strange one as when we checked the http.sys cert bindings everything looked correct except that the AD FS Hostname:port binding did not have a Ctl Store defined:-

Of interest here is that the Application ID is the AD FS 2012 R2 SSL Certificate Owner Application ID so on the surface does look to have been created by AD FS. We did not manage to identify why the CTL Store was not correctly set in this case. The PowerShell script provided in the ‘Issue Detection PowerShell Script’ section below can be run on an AD FS 2012 R2 server to determine if this issue is present.

Root Cause 2 - Resolution

In this case we can use the following PowerShell cmdlet to regenerate the ADFS Certificate bindings on the AD FS server:-

Set-AdfsSslCertificate -Thumbprint 3638de9b03a488341dfe32fc3ae5c480ee687793

The Thumbprint here is the Thumbprint for the ADFS Server SSL certificate and needs to be set accordingly. This cmdlet should be run on each AD FS server in a farm. For further details on this cmdlet please see the following article - http://technet.microsoft.com/en-us/library/dn479374.aspx 

Root Cause 3 – CA issued server certificate placed in the AdfsTrustedDevices store

The AdfsTrustedDevices store is an AD FS specific store and should not need to have any certificates manually placed in here.

We found in one case that the ADFS service SSL certificate was incorrectly placed in the AdfsTrustedDevices store as well as the correct Local Machine, Personal store. The AdfsTrustedDevices store should only contain the MS-Organization-Access certificate which is the self-signed cert used for issuing Workplace Join certificates, and the Proxy Trust certificates for each of the Web Application Proxy servers.

As detailed earlier, having a CA Issued certificate in a store where only Self-Signed certs would normally exist affects the CTL generated from this store and the CTL will then only contain the CA Issued certificate. The presence of a CA issued certificate in the AdfsTrustedDevices store changed the CTL being returned and led to the proxy trust failure.

The PowerShell script provided in the ‘Issue Detection PowerShell Script’ section below can be run on an AD FS 2012 R2 server to determine if this issue is present.

Root Cause 3 – Resolution

Deleting the non-self signed SSL Server Certificate from the AdfsTrustedDevices store resolved the issue and meant that the CTL being generated from the AdfsTrustedDevices store was now correct.

Root Cause 4 – Proxy Trust certificate propagation issues across an AD FS 2012 R2 farm

When a Proxy Trust is established with an AD FS server, the Proxy Trust certificate is written to the AD FS configuration database and added to the AdfsTrustedDevices store on the AD FS server that handled the Proxy Trust set-up. In an AD FS farm a regular task runs on each AD FS server in the farm which polls the AD FS configuration store for the current list of Proxy Trust certificates and updates the local AdfsTrustedDevices store.

We have recently discovered an issue where this task only successfully runs if Device Registration has been initialized.

If Device Registration has not been initialized then Proxy Trust certificate propagation across the AD FS farm may fail and a Proxy Trust will only work against the AD FS server the trust was established with. As the CTL is based on the AdfsTrustedDevices store, if this has not been populated with the correct Proxy Trust certificates and a Web Application Proxy server talks to an AD FS server other than the one it established the proxy trust with, then the proxy trust may fail due to the CTL being incorrect.

You can check if Device Registration has been initialized by running the following PowerShell cmdlet on one of the AD FS servers:-

                Get-AdfsDeviceRegistration

This requires Enterprise Administrator credentials to run. If Device Registration has been initialized you would see output similar to the following:-

The PowerShell script provided in the ‘Issue Detection PowerShell Script’ section below can also be run on an AD FS 2012 R2 server to determine if this issue is present.

Note – if your load balancing for your AD FS server farm uses client IP based affinity we would generally expect a Web Application Proxy server to use the same AD FS server other than in a failover scenario.

Root Cause 4 - Resolution Options

Update 1st July 2014 - A fix is now available for this specific issue in the June 2014 Windows update rollup - http://support.microsoft.com/kb/2964735

1. Initialize Device Registration in Active Directory

The current recommended resolution for this is to initialize Device Registration using the following PowerShell script on one of the AD FS 2012 R2 servers which creates the DRS containers in Active Directory.

Initialize-ADDeviceRegistration

Note – this needs to be run with Enterprise Administrator credentials and requires that the Windows 2012 R2 AD schema update has been deployed. For further details on this cmdlet please see the following article - http://technet.microsoft.com/en-us/library/dn479332.aspx

This cmdlet does *not* enable the device authentication nor the device registration service in the ADFS servers. Enabling Device Registration is a second step in the DRS set-up process.

The AD FS Product Group are currently investigating a fix for this issue but at the time of writing we do not have further details on this or any expected timelines. We will provide an update when we have further information on this.

2. Manually Sync Proxy Trust Certificates using PowerShell

The PowerShell script provided in the ‘Issue Detection PowerShell Script’ section below can be run with the - syncproxytrustcerts switch to sync down missing  Proxy Trust certificates from the AD FS configuration to the AdfsTrustedDevices certificate store. This script should be run on each of the AD FS servers in the farm.

Note – this is not a permanent solution as the Proxy Trust certificates will be renewed on a regular basis so until a fix is available Option 1 is the recommended resolution.

3. Use source IP based affinity on your AD FS 2012 R2 load balancing solution

Using source IP based affinity on the AD FS Load Balancer will mean a Web Application Proxy server is generally going to use the same AD FS server it established the initial trust with and therefore not be reliant on the proxy trust certificate being propagated to other members of the AD FS farm. If the preferred AD FS node a Web Application Proxy server changes or in a failover situation then this issue may still be exposed. 

Root Cause 5 – System time mis-match between WAP and ADFS

It is expected that the Web Application Proxy server and AD FS server are time synchronized. Where Web Application Proxy is not domain joined this may not be the case and we have seen Proxy Trust issues as a result of this e.g.

AD FS Server time = 12pm GMT timeone

Web Application Proxy Server time = 12pm PST timezone (8pm GMT)

In this scenario while the system time may seem synchronized the timezone difference means that the Web Application Proxy server is 8 hours ahead of the AD FS server time. When the Proxy Trust relationship is established, the self-signed Proxy Trust certificate generated by the Web Application Proxy is not yet valid on the AD FS server and so the Proxy Trust fails.

While this is not directly related to CTL issues it’s one to be aware of with similar symptoms.

Root Cause 5 - Resolution

Ensure that the system time of the Web Application Proxy server and AD FS 2012 R2 server are in sync including that the timezones are set appropriately. 

Root Cause 6  – SSL termination taking place on a network device between Web Application Proxy and AD FS server

We saw a recent issue where we were seeing the following in the AD FS event logs on the Web Application Proxy server and AD FS server.

In the AD FS event log on the Web Application Proxy server:- 

In the AD FS event logs on the AD FS server:-

There are a number of things to note here:-

1. The AD FS server was presented a client certificate. This is different to most of the above issues where a client certificate is not seen at the AD FS server.
2. The thumbprint seen at the AD FS server is different to the one seen in the Web Application Proxy AD FS event logs.
3. The Subject Name seen by the AD FS proxy is not what we would expect from a proxy trust certificate which should have the form 'ADFS Proxy Trust - webappproxymachinename'

Given the difference in Thumbprint and unexpected Subject Name we investigated the network devices between the Web Application Proxy server and the AD FS server and determined that the hardware load balancer sitting in-front of the AD FS farm was carrying SSL bridging i.e. terminating the SSL connection from Web Application Proxy and establishing a new SSL connection to the AD FS server. The certificate seen by the AD FS server was being presented by the hardware load balancer.

As the Proxy Trust relationship between Web Application Proxy and AD FS server is client certificate based this was breaking the proxy trust relationship.

The SSL communications between the Web Application Server and AD FS server cannot be pre-terminated on a load balancer, Firewall, other reverse proxy device and doing so will break the proxy trust relationship. 

Root Cause 6 - Resolution

Disabling SSL Termination on the hardware load balancer to allow the SSL negotiation to happen between Web Application Proxy and AD FS server resolved this issue.

Issue Detection PowerShell Script

The following PowerShell script can be run on an AD FS server to determine if any of the first 4 issues detailed above are present. In an AD FS farm the script should be run on each farm member.

The script can also be run with the –syncproxytrustcerts switch to sync down missing Proxy Trust certificates from the AD FS configuration to the AdfsTrustedDevices certificate store as a temporary resolution for Issue 4. This sync would need to be done on each AD FS server.

Summary

First of all, congratulations if you made it this far :-)

One the key things we wanted you to explain is that the Proxy Trust relationship is based on Client Certificates and can be affected both by http.sys bindings and the associated Ctl Store on the AD FS server. Having a specific IP:port binding takes precedence over the ADFS Hostname:port binding and may lead to the wrong Certificate Trust List being sent. The wrong CTL will prevent Web Application Proxy from presenting the correct Proxy Trust certificate.

The first place to look when troubleshooting a Proxy Trust issue is the http.sys certificate bindings on the ADFS server by running the following command from the command prompt:-

                netsh http show sslcert

Resolution steps will depend on what bindings are found and on why these are there. The simplest solution is to ensure that there is not a specific IP:port binding in place for the IP address that the AD FS Service FQDN resolves to.

Another item to check is the contents of the AdfsTrustedDevices certificate store for the presence of both the expected Proxy Trust certificates and for any unexpected CA issued certificates.

We also covered a number of other causes of Proxy Trust failures that we've encountered with our customers and provided a PowerShell script that can be run on an AD FS 2012 R2 server to detect a number of the issues detailed in this article.

We hope this helps some of you solve Proxy Trust issues you might come up against and we’d love to hear any other conditions you might come across that lead to similar issues.

Update 1st July 2014 - article updated with hotfix availability for Issue 4.

Ian Parramore, Senior Escalation Engineer, Web Application Proxy Support team             

Thanks to the following for their input on this article:-

Billy Price, Senior Escalation Engineer, Web Application Proxy Support team

Meir Mendelovich, Product Manager, Web Application Proxy Product Group

Ramiro Calderon, Software Development Engineer, AD FS Product Group

Mike Bishop, Product Manager, http.sys Product Group

Andrew Rockall, Software Development Engineer, http.sys Product Group 

 We are happy to announce that Web Application Proxy documentation is now available in many more languages:

We are committed to evolve and improve Web Application Proxy all the time. Shortly after the release, we have identified that maintaining the certificates that validate ADFS tokens in Web Application Proxy can be difficult and prone to error. Therefore we have provided an update that is rolling to all machines running Web Application Proxy using Windows Update as part of May 2014 update - KB 2935608 (http://support.microsoft.com/kb/2935608).

If your machine is up to date, it will no longer display the ADFSTokenSigningCertificatePublicKey and will not let you update it. You would notice a message stating “WARNING: The parameter ADFSTokenSigningCertificatePublicKey is obsolete and will not be set”.

Warning received when trying to update the parameter:

Web Application Proxy configuration output shows the parameter as obsolete:

Before you spend any additional time trying to track down a problem certificate, let me assure you that this is not a sign of any issue with your WAP deployment. The message is merely stating that this parameter is obsolete, not the certificate.

How about a little background…

Web Application Proxy keeps track of the AD FS Token-signing certificate public key to verify that a client authentication token was issued and signed by the correct trusted AD FS farm. Initially, WAP only obtained the public key once during the initial post-install configuration. If you do not have this update installed then you can view the key stored in the WAP configuration and this will match the public key from the Federation Metadata xml.

This static key storage is not a problem unless the AD FS farm is enabled for automatic certificate rollover. With this configuration, the Token-signing certificate is updated after the defined certificate duration period (default : 1 year). When the AD FS Token-signing certificate updates, the WAP server will no longer be able to verify the client authentication token was issued from the trusted AD FS farm. The AD FS administrator may also update the Token-signing key manually which will have the same affect, any change of this key will result in a problem with WAP. The WAP server will not be able to validate the token signature and client access to published applications will fail. You may see this warning in the Web Application Proxy events: 

Warning Event 13012 – Web Application Proxy received a nonvalid edge token signature.

Prior to this recent update, the Web Application Proxy storage for the Token-signing key would require a manual update via PowerShell if needed. With the update applied, the Web Application Proxy will monitor the FederationMetadata and automatically retrieve the updated Token-signing key when it changes.

This is why the ADFSTokenSigningCertificatePublicKey parameter is now obsolete. The Token-signing key information is automatically updated and no longer requires a manual reset.

Billy Price, Senior Escalation Engineer, Web Application Proxy Support team

Hi,

Azure AD Application Proxy preview is now live and available for you to play with.

See all the details here: http://blogs.technet.com/b/ad/archive/2014/06/10/public-preview-of-azure-ad-application-proxy.aspx

The documentation is here: http://msdn.microsoft.com/en-us/library/azure/dn768219.aspx

This is a big step for the application proxies. We now provide a comprehensive conditional access solutions deployed on-premises and in the cloud. We will post here more details on this new service and explain how it revolutionize remote access.

The nice thing about cloud services is that we can constantly update them and add new features. We will use this blog to notify you on new feature availability. Keep in mind that the current preview is only a baseline and we will extend and improve it over the coming weeks and months.

We will be more than happy to hear your feedback. You can use this blog comments or mail us at AadapFeedback @ Microsoft.com.

Meir Mendelovich, Senior PM, Active Directory product group

Hi,

It is very nice to see that people in the industry start to understand the revolution that Azure Active Directory Application Proxy brings: http://www.networkworld.com/article/2361973/infrastructure-management/microsoft-azure-service-lets-it-control-identity-and-access-management.html

We see in our dashboards that many of you are already trying it. We see connectors being downloaded and registered, applications being published and on-going usage. Don't hesitate and utilize it as much as you want. Internally, we publish on our test tenants public internet sites like bing.com and our entire team is using the URL that is exposed by the proxy to generate real-world load.

If you haven't already done this, go to your Azure AD tenant and try it by yourself.

Meir Mendelovich, Active Directory product group

http://www.networkworld.com/article/2361973/infrastructure-management/microsoft-azure-service-lets-it-control-identity-and-access-management.html

Welcome back everyone. In this blog we’re going to look at SNI which is an area we’re seeing a number of questions on so wanted to spend a bit of time explaining.

Server Name Indication (SNI) is a feature of SSL TLS and both Web Application Proxy and AD FS 2012 R2 use it to enable simpler deployment and remove networking prerequisites. Some client applications don’t support SNI but there is an easy way to workaround this using a fallback certificate. In this blog we will explain what SNI is, how it works and what to do if you have non-SNI capable clients.

What is this SNI of which you speak?

SNI is an extension to the TLS SSL protocol that allows the client to include the Hostname the client is connecting to in the SSL Client Hello. A server can then use the SNI header to determine which certificate to serve to the client. A key benefit of SNI is that is allows a server to host multiple certificates on the same IP/port pair instead of needing an IP per certificate (assuming you are using port 443).

SNI relies on the client supporting SNI and sending the Server Name extension in the SSL Client Hello. If the SSL Client Hello does not contain the SNI header then http.sys is unable to determine which certificate to serve and will reset the connection.
While most clients do support SNI you may encounter non-SNI capable clients that you need to provide support for. Some non-SNI clients we have seen issues with include:-

-    Older browser versions or legacy operating systems (Internet Explorer on Windows XP, Mozilla prior to v2.0, Safari prior to v3.0 Mac OS X 10.5.6, etc.)
-    Some older versions of WebDAV
-    ActiveSync for Android
-    Internet connected devices without recent OS/TLS updates (VOIP conferencing devices, proxy devices, etc.)

We have also seen SNI related issues with various Hardware Load Balancers health check requests not sending the SNI header although this is often due to the necessary configuration not having been put in place. We’ll be covering Hardware Load Balancer health check requirements and other considerations in more detail in an upcoming blog.

If you want the full technical details on SNI you can read the relevant RFC - https://tools.ietf.org/html/rfc3546#section-3.1

How to tell if the client is sending an SNI header?

If you suspect an application is having connectivity issues through Web Application Proxy or to an AD FS 2012 R2 server a Network Monitor trace of the traffic either on the client or on the Web Application Proxy / AD FS is a good way to determine if an SNI header is being sent and, if you are getting a RESET back. The following is an SSL Client Hello taken from a Network Monitor trace with the Server Name (SNI) header present in the Client SSL Hello:-

If the client supports SNI then all will be good and the standard Web Application Proxy and AD FS Server 2012 R2 certificate bindings will be fine. If the client does not send an SNI header in its SSL Client Hello then http.sys will send a TCP RESET on the connection. The following shows the network traffic you would see if this is the case:-

Supporting Non-SNI Capable Clients

As mentioned above, if you are having connectivity issues from a specific client application and you suspect the client may not be SNI capable, running a Network Monitor trace on your Web Application Proxy / AD FS 2012 R2 server while the client attempts to connect will show you if the client is sending an SNI header in the SSL Client Hello.
If you determine you have issues with non-SNI capable clients you have a few different options:

-    Do nothing. It may be that you make a deliberate decision not to support that browser/operating system/client and a valid choice is to decide not to support non-SNI clients
-    Upgrade the client to an SNI capable version. This may be the browser, OS, device firmware etc. An upgrade should be considered if possible.
-    Enable/configure SNI support on the client. Some clients like hardware load balancer health checks do support SNI but sometimes it is off by default.
-    Configure a fallback certificate in http.sys – see next section

Configuring an http.sys fallback certificate

Our current recommendation if you need to support non-SNI capable clients is to add a manual http.sys certificate binding for 0.0.0.0:443 using the following command:-

netsh http add sslcert ipport=0.0.0.0:443 certhash=certthumprint appid={applicationguid}

This configures http.sys with an IP Wildcard certificate binding which will be used as a fallback certificate. This would need to be run on each server in a farm.

When running the ‘netsh http add sslcert’ command the certhash parameter would be the certificate thumbprint for the certificate you want to use as the fallback cert. Please see the ‘How to determine a certificate’s thumbprint’ section below for additional details.

The appid parameter is the GUID of the application that created the binding but can also be manually set.  The following are the 2 application ID’s for Web Application Proxy and the AD FS service and using either of this would be fine:-

    {5d89a20c-beab-4389-9447-324788eb944a} – AD FS App ID
    {f955c070-e044-456c-ac00-e9e4275b3f04} – Web Application Proxy App ID

The command you would run would therefore look something like this:-

netsh http add sslcert ipport=0.0.0.0:443 certhash=3638de9b03a488341dfe32fc3ae5c480ee687793 appid={5d89a20c-beab-4389-9447-324788eb944a}

Fallback certificate considerations

As you can only configure a single certificate as the fallback cert it is important that the certificate you use covers all the FQDN’s that you need to provide SNI fallback support for which may include:-

-    AD FS service FQDN e.g. adfs.contoso.com
-    Enterprise Registration (Workplace Join) FQDN’s e.g. enterpriseregistration.contoso.com
-    Any Web Application Proxy published application External FQDN’s e.g. sharepoint.contoso.com

Using either a Subject Alternate Name certificate or a Wildcard certificate is supported. A Wildcard certificate may give greater flexibility when adding further published applications in the future.

Note - while using a fallback certificate is certainly going to be needed to support some clients and scenarios, please be aware that this is a manual binding that you are going to have to remember to maintain and update on each node as certificates expire and are replaced.

Can I use an IP Specific fallback binding?

The simple (and recommended) answer to this is No.

However, as with most things the reality is a bit more complicated.

http.sys does allow you to configure IP specific bindings e.g. 1.2.3.4:443 but you need to be very careful about going down this route as http.sys in Windows 2012 R2 uses the following order of precedence when determining which certificate to serve:-

If an IP:Port binding is configured then, as per the above order of precedence, the IP specific binding will take precedence over all other bindings so any SSL traffic to that IP address will receive the certificate specified in the IP:Port binding.

Important: An IP specific binding is therefore not a fallback certificate – it becomes the exclusive certificate on the specified IP address and takes precedence over SNI bindings. 

This can therefore break any traffic relying on the SNI bindings and in particular, you risk breaking the Certificate Trust List used by AD FS 2012 R2 as this is defined in the AD FS SSL Service certificate binding.

Our advice if you do need to use an IP specific binding is as follows:

-    Do not use an IP Specific binding on the primary IP address of Web Application Proxy or AD FS 2012 R2. The primary IP address should be used for the majority of traffic and using SNI

-    If required, configure a fallback cert on 0.0.0.0:443 with a certificate that covers your AD FS service FQDN, EnterpriseRegistration FQDN’s and other Web Application Proxy published application External FQDN’s using a Subject Alternate Name or Wildcard certificate

-    If you need to use a different certificate for a specific FQDN then add a 2nd IP address to the Web Application Proxy server and use an IP specific certificate binding on this secondary IP. This certificate binding will then be exclusively used for all traffic to this IP without affecting traffic to the primary IP address and SNI bindings. For each additional certificate required add an additional IP address and bind the certificate to this.
You then just need to ensure your inbound traffic for the relevant service FQDN is sent to this secondary IP address either through DNS resolution or load balancer / firewall configuration.

Note: From an AD FS 2012 R2 perspective we would not expect an IP specific binding to be needed as you can just configure your AD FS SSL Service certificate as the 0.0.0.0:443 fallback cert.

Our general advice is to avoid IP specific bindings due to the complexities these introduce and the fact that you’re very much moving into a manual certificate binding management situation. It can be done but you need to understand the above points to avoid hitting issues. We expect that using a 0.0.0.0:443 fallback binding will solve the majority of scenarios.

How to determine a certificate’s thumbprint

There are many different ways to determine a certificates thumbprint. Here are few different options depending if you prefer scripting or GUI:-

1) Look at the Web Application Proxy application configuration

You can run the following PowerShell command on a Web Application Proxy server to identify the Certificate thumbprints in use for each of the Web Application Proxy publishing rules:-

Get-WebApplicationProxyApplication | fl Name, ExternalUrl, ExternalCertificateThumbprint

Sample output:-
Name                          : ClaimsApp
ExternalUrl                   : https://claimsapp.contoso.com/claimapp/
ExternalCertificateThumbprint : 1D96CD2450B01E5660F756D1BDDE3B49B4887035

2) Use PowerShell to review the installed Certificates on the server

You can run the following PowerShell command on a server to return the certificates installed in the Local Computer Certificate store :-

dir Cert:\LocalMachine\My

Sample output:-

Thumbprint                                Subject
----------                                -------
1FAAA1B7FA33D266E5CC54E4EBE41CB6C68E583B  CN=ADFS ProxyTrust - 2012R2-WAP01
1D96CD2450B01E5660F756D1BDDE3B49B4887035  CN=*.contoso.com

3) Review the existing http.sys bindings

You can check existing certificate bindings using the following command to identify the thumbprint of an existing binding already using the target certificate:-

    netsh http show sslcert

The following is an example of a binding for a Web Application Proxy published application:-

    Hostname:port                : claimsapp.contoso.com:443
    Certificate Hash             : 1d96cd2450b01e5660f756d1bdde3b49b4887035
    Application ID               : {f955c070-e044-456c-ac00-e9e4275b3f04}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Disabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled
   
4) Use the Certificate MMC to look at the Certificate properties

You can also open the Certificate MMC, Local Machine, Personal store and open the relevant cert from there to identify the certificate thumbprint:-

Summary

Server Name Indication (SNI) is used both by Web Application Proxy and AD FS 2012 R2 in their http.sys certificate bindings. For Web Application Proxy in particular this allows multiple, disparate certificates to be hosted on a single IP/port and allows Web Application Proxy to publish multiple sites without the need for multiple IP address.

SNI relies on the client sending the Server Name header in its SSL Client Hello. If this is not supplied in the client TLS negotiation then http.sys will reset the connection.

If you need to support non-SNI capable clients then it is possible to bind a fallback certificate in http.sys to 0.0.0.0:443. This certificate will be served in the server hello to any non-SNI capable clients. As you can only have one fallback certificate bound to 0.0.0.0:443 this certificate should include all the necessary FQDN’s you need the fallback certificate to support.

It is possible to use IP specific bindings but these take precedence over SNI bindings and should be used with caution.

As always, let us know if you have any feedback or comments and if you have any requests for specific articles and we’ll see what we can do.

Ian Parramore, Senior Escalation Engineer, Web Application Proxy support team

Check out our latest addition to TechNet: Web Application Proxy Troubleshooting!

Let us know if there's anything else you'd like to see there!

We’ve had a number of questions around using SSL Termination with Web Application Proxy and AD FS 2012 R2. In this blog we’ll look at when you can and can’t use SSL Termination and some of the considerations around this.

What do we mean by SSL Termination?

SSL Termination (or SSL Bridging depending on your preference) is where a device between a client and a target server terminates the SSL connection in front of the target server and then sets-up a new connection to the target server. This behavior is commonly seen with Network Load Balancers, Reverse Proxy servers and other devices with an interest in inspecting and accessing the application data being sent over the SSL connection.

This might be done to make traffic routing decisions, add HTTP headers with additional information to the request, carry out content inspection etc.

Note - we are specially talking about termination of the SSL traffic not the underlying TCP/IP connection.

So, can I use SSL Termination with Web Application Proxy and AD FS 2012 R2?

To bridge or not to bridge, that is the question, and there are 3 answers to this – Yes, No and Maybe :-)

As with all these things the answer is really “It Depends” and we have 3 main areas of consideration here:-

SSL Termination between	Can you use SSL Termination
Web Application Proxy and ADFS 2012 R2	No
Client and Web Application Proxy / AD FS Proxy*	Maybe
Web Application Proxy and Published Web Server	Yes

* a Web Application Proxy server also performs the AD FS Proxy role

Let’s look at each of these in turn to understand why we can/can’t use SSL Termination and the reasons behind this.

1)    SSL Termination between Web Application Proxy and AD FS 2012 R2

There is no discussion or debate on this one – it’s a definite No.

The reason for this is because the Proxy Trust relationship between Web Application Proxy and AD FS 2012 R2 is based on Client SSL Certificates.

A Client SSL Certificate is only available to the endpoint where the SSL Connection is established. When you terminate SSL in front of the AD FS 2012 R2 server, the AD FS server no longer sees the Web Application Proxy server Proxy Trust certificate breaking the authorization channel between the two. This will lead to Event ID 442’s and issues similar to those discussed in the following blog:-

http://blogs.technet.com/b/applicationproxyblog/archive/2014/05/28/understanding-and-fixing-proxy-trust-ctl-issues-with-ad-fs-2012-r2-and-web-application-proxy.aspx

2)    SSL Termination between Client and Web Application Proxy / AD FS Proxy

This scenario is a little less clear although strictly speaking from a product perspective the answer is No. The reason for this is that some product features will break if you terminate SSL in front of Web Application Proxy server. Specifically the following features will not work:-

-    Workplace Join / Device Registration
-    Client SSL Certificate authentication

As you may have guessed, these features rely on Client SSL Certificate negotiation which, as we already discussed, will break if the SSL Connection is terminated in front of the Web Application Proxy server.

If you do not plan to use the above features, then terminating SSL in-front of Web Application Proxy should be fine, although we’d caveat that with the fact that this is not a scenario that we have tested to any degree of depth.

If you do terminate SSL in-front of the Web Application Proxy server then you have some added complexity in whether your terminating device (Hardware Load Balancer, Reverse Proxy etc) sends an SNI header when it makes the SSL Connection to the Web Application Proxy server.

An SNI header should be sent in the SSL Server Hello and this should match the external FQDN of the published application. The following blog talks in a lot more detail about what SNI is, why it’s important and workarounds to support non-SNI clients:-

http://blogs.technet.com/b/applicationproxyblog/archive/2014/06/19/how-to-support-non-sni-capable-clients-with-web-application-proxy-and-ad-fs-2012-r2.aspx

The following F5 article is also a good reference in terms of how to configure an F5 hardware Load Balancer to send an SNI header when SSL Termination is in use:-

https://devcentral.f5.com/articles/big-ip-and-adfs-part-5-working-with-adfs-30-and-sni#.U4imenkU-Uk

As you can see, using SSL Termination in-front of Web Application Proxy reduces functionality and introduces complexity but, if you are aware of the issues it can work and can bring some benefits e.g. a Hardware Load Balancer can add x-forwarded-for header to inbound client HTTP request.

3)    SSL Termination between Web Application Proxy and Published Web Servers

There are currently no issues we are aware of with using SSL Termination between Web Application Proxy and published Web Servers so at this point in time it’s a Yes :-).

As mentioned above the main issues with SSL Termination is the impact it has on Client SSL Certificate usage. As Web Application Proxy is acting as a Reverse Proxy for the published server any Client SSL Certificate will be not seen by the published Web Server so, as long as the device carrying out the termination, does not adversely interfere with the HTTP traffic things should work fine.

If the Web Server is using SNI based certificate bindings then you would also need to be aware of the SNI considerations discussed above when the device makes the onward SSL connection to the target web server.

Summary

SSL Termination / SSL Bridging is a commonly used configuration especially with Hardware Load Balancers.

There are some scenarios where using SSL Termination will definitely break Web Application Proxy / AD FS 2012 R2 functionality. Specifically, when SSL Termination is used between the Web Application Proxy and AD FS 2012 R2 servers it will break the Proxy Trust relationship.

It can also leads to a reduced level of functionality when used between the clients and Web Application Proxy server although, if you do not need features that are Client SSL Certificate based such as Workplace Join or Client SSL Certificate authentication, then SSL Termination in front of Web Application Proxy can work OK.

If you do carry out SSL Termination in-front of the Web Application Proxy server then you need to understand your devices SNI capabilities when it establishes the new SSL Connection to the Web Application Proxy server.
SSL termination between the Web Application Proxy server and the published web servers should be fine and not cause any issues assuming that the terminating device does not adversely interfere with the HTTP traffic and any SNI requirements are met.

As always, let us know if you have any comments or questions. We’re currently working on a Hardware Load Balancer health check blog at the moment to follow up on this one.

Ian Parramore, Senior Escalation Engineer, Web Application Proxy support team

Application Proxies are a very sensitive point in the organization, being the gatekeepers for most of the organization sensitive applications. Not only they are exposed to the Internet, they are also well known entry points. This is why we take security very seriously. In fact, this was one of the reasons we decided to build both Web Application Proxy and Azure Active Directory Application Proxy from scratch with a fresh code base.

Here are some security measures that apply to both solutions:

Preauthentication is the best gatekeeper

The best method to keep the bad guys out and protect your applications from various application attacks is implementing preauthentication. It means that users can access the application only after they were authenticated. The main role of the proxy is to enforce this preauthentication. This puts a very high bar on most attacks and is a strong mitigation for distributed denial of service (DDoS) attacks impact on the applications. In case DDoS attacks happen, only the authentication service that is dedicated for external users is bombarded, and the backed applications are not impacted.

Cleaning the HTTP and SSL traffic before it arrives to the application

A proxy, by its nature, protects the backend application from various attacks just by terminating the external SSL and HTTP session and starting another one with the backend application. This practice eliminates all layer 2 and 3 attacks. A recent example for such attack is the Heartbleed bug that exposed all applications running a specific SSL stack to attacks that read server memory by sending wrong data in the SSL protocol. If such application is protected by a proxy, this illegal request will never reach its server as the proxy will drop such SSL sessions.

It also ensures that the HTTP request is validated and well-formed before the request is passed to the published server.

Reducing the attack surface

The first step in good security practices is to reduce the attack surface. For us this means that we are exposing to the internet only to the bare minimum that is required to provide a service.

In Web Application Proxy, we are doing this by running the proxy directly on top of Windows HTTP.SYS layer and utilizing its capabilities to assure that all other traffic will be filtered by the Windows networking stack. We instruct HTTP.SYS to accept only traffic to the very specific URLs that are published. For example, if the admin decided to publish with external URL https://sp.contoso.com/site3 requests to other paths of this domain or to other domains will be filtered by HTTP.SYS and will not arrive to the proxy.

When the Azure Active Directory Application Proxy is used, all these requests to unpublished resources are filtered on our cloud service and will never reach your corporate network.

Running low, as low as possible

As another layer of protection, the proxy that is exposed to the internet runs the on-prem components in the context of an account called “network service”, which has very few privileges. This assures that even if an attacker somehow manage to run code in the context of the proxy service in Web Application Proxy or the Azure AD Application Proxy connector, it will have very little impact on the machine. This is why Web Application proxy is composed of two different services, one that is exposed to the internet and has very limited privileges and another one that is not exposed to the internet and performs actions that require more privileges like updating configuration or listening for a new URL.

Never store passwords

An important security measure that is used to reduce the impact of attacker that gains full access to the proxy or connector machines is to make sure that it does not store passwords, not on the disk and not in memory. This is very different than the security practices of the past where edge components stored passwords in memory and in many cases also on the disk. The Application Proxies can achieve this because of the separation between the proxy functionality and the authentication and authorization that are done on AD FS and Azure AD.

Microsoft Security Development Lifecycle

Other than the issues described above there are hundreds of less prominent techniques and guidelines that we used when we developed the application proxies to assure security. Most of them derive from Microsoft Security Development Lifecycle (SDL), a process and framework that is used in Microsoft to develop more secure products and to share the experience we gain. The SDL Web site contain many more details on the value gained by using this practices.

Azure Active Directory Application Proxy: taking security to a new level

Azure Active Directory Application Proxy takes security to the cloud scale. Its security model is even more robust than the traditional on-premises model. There is no traffic that is inbound to the organization. Everything is sent to the cloud. This means that all filtering and protection occurs in the cloud, only authenticated and valid traffic arrives to the corpnet.

Other than the benefit of keeping the bad traffic off the corpnet, the cloud approach also has the advantage of large and elastic scaling. It is one thing to knock down two machines on the DMZ with DDoS attack but it is a completely different story to knock down Azure Active Directory service that runs in much bigger scale. This again provides a strong level of protection for the published on-premise services and infrastructure.

Another benefit of the cloud is the ability to quickly update in case of vulnerability. We make sure that the cloud service is running on the most up-to-date networking stack and it is patched immediately if something is detected. All of this happens behind the scenes so there is absolutely no effort needed from the organization.

For more information on Azure Active Directory Application Proxy please see the following link: http://msdn.microsoft.com/en-us/library/azure/dn768219.aspx

I LOVE POWERSHELL

Web Application Proxy LOVES POWERSHELL

For us PowerShell is the fundamental API to our system. This is how the UI works. This is how our remote management work. This is how we test the product. This is how we deploy it in our own labs.

As you probably noticed, the UI wizards always end by showing you the PowerShell command it sends to apply the changes. This is a great way for you to learn the basics, how to start doing the regular stuff. But, we have more for you. Here are some advanced commands and how you can use them with the PowerShell goodies to better manage your Web Application Proxy deployments.

To start, here are the commands aliases that allow much shorter and more readable scripts:

Now let’s see the most common PowerShell tricks using the standard cmdlets:

As Web Application Proxy is a standard Windows Server role service, you can use many Windows Server PowerShell tools to control Web Application Proxy:

And finally, here are some tricks for managing Web Application Proxy multi-machine deployments:

$Author.Name =“Meir Mendelovich”

$Author.Role =Microsoft.ProductGroupTitles.SeniorProgramManager

CLS

We’ve recently resolved a couple of instances of a very specific Web Application Proxy post-install configuration failure. The conditions under which you would see this are quite specific but if you do encounter this it’s not a particularly easy problem to either identify or resolve so we wanted to share our findings to hopefully save some of you some time and pain.

What are the Symptoms of the Issue?

After installing the Web Application Proxy feature you then need to run through the Web Application Proxy post-installation configuration wizard. After completing all the necessary details the post-install configuration tasks will then run.

Under certain conditions which will come to a bit later, retrieving the ADFS Proxy configuration may fail with the error:-

“AD FS Proxy could not be configured. Unable to retrieve proxy configuration data from the Federation Server.”

You will also see the following logged in the AD FS event log on the Web Application Proxy server:-

What’s going on in the Background to cause this?

As part of responding to the AD FS Proxy configuration request the AD FS server has to determine if Device Registration has been initialized and enabled. This is so the AD FS Proxy service on the Web Application Proxy server knows whether it needs to enable the Device Registration endpoints and certificate bindings,and for which UPN suffixes.

As part of the DRS discovery logic, if the DRS configuration objects which are stored in Active Directory, cannot be found in the local domain, the AD FS 2012 R2 server will try to connect a DC in the root domain to enumerate all GCs under the CN=Sites container and that host the root domain. From this list, it iterates over each GC until it finds a reachable GC to try to locate the DRS configuration.

In a complex, segregated network environment the AD FS server may not have network access to many of these Global Catalog servers which leads to delays while the TCP connection attempts to these GC servers time out.

The Web Application proxy server call to retrieve the AD FS Proxy configuration has a 100 second timeout so if the cumulative delays due to the GC network timeouts exceeds this timeout the post-install configuration will fail. Web Application Proxy will retry the configuration call a number of times but, if you are seeing this issue, each call will be subject to the same delays and will fail.

We are aware that this behavior may cause issues in complex network environments and we are working with the AD FS Product Group to review this and look at how we can modify this behavior moving forward but that’s not going to help you if you’re hitting this issue now.

How can I check if I am hitting this issue?

The first thing to check for is the presence of the “System.Net.WebException: The operation has timed out” Event ID 422 error in the AD FS event logs on the Web Application Proxy server. This is the primary indicator that you may be hitting this problem.

If you believe you may be hitting this issue you can also run a Network Monitor trace (or your network capture tool of choice) trace on the ADFS 2012 R2 server while attempting to run the Web Application Proxy post installation configuration.

Network Monitor 3.4 can be downloaded from the following link - http://www.microsoft.com/en-us/download/details.aspx?id=4865

You can then apply the following display filter in Network Monitor - property.TCPSynRetransmit

This filter will show re-transmitted TCP SYN packets.

A TCP SYN is the first packet used to establish a TCP connection. Re-transmitted SYN’s means that the expected SYN ACK response was not received so the OS retries the SYN packet. You can think of this a bit like trying to phone someone, not getting an answer, waiting a few minutes and then trying again.

We’re specifically interested in unanswered SYN packets to port 139 and 445 which are the ports that will be used to talk to the Global Catalog server.

If you hitting this issue you would see something like the following with the above filter applied.

Note – as the Web Proxy Application server will make a number of attempts to retrieve the configuration you will see multiple cycles of attempts. For the purposes of solving this issue we just need to bring one cycle under the 100 second timeout so we’re interested in unique Global Catalog servers and not how many times we try to connect to them.

A Short Diversion into TCP/IP SYN Re-transmissions

The TCP/IP SYN re-transmission behavior is very specific and worth a short discussion on as tuning this does provide a possible workaround for the issue.

As noted above if a TCP SYN connection attempt does not receive a response it will re-transmit the SYN attempt and there are a couple of key parameters that control the behavior here:-

- TcpMaxConnectRetransmissions – default value 2 – this controls how may times the TCP stack will retry the initial SYN packet if a SYN ACK response is not received. The following KB talks in more details about this setting - http://support.microsoft.com/kb/2786464

- InitialRTO – default value 3000 msec – this is the initial retransmit timeout period and gets doubled between each successive retry. The following KB talks in more details about this setting - http://support.microsoft.com/kb/2472264

For example, with default settings you will see the following behavior:-

Syn Attempt	Timeout Period	Cumulative Time
Initial SYN	3 seconds	3 seconds
1st SYN re-transmit	6 seconds	9 seconds
2nd SYN re-transmit	12 seconds	21 seconds

These settings are relevant as it gives us a means to control how long the timeout takes for each unavailable Global Catalog server.

Potential Workarounds

Now that we know what’s causing the timeout let’s look at potential workarounds for this.

1) Allow the Network Traffic to one of the Global Catalog servers

It’s easy for me to sit here and write this but I’m not the one who has to go and talk to the network/Firewall guys and convince them :-)

Seriously though, if you are able to open either TCP port 139 or 445 to one of the first Global Catalog servers that AD FS can’t reach this should resolve the issue as the AD FS server would stop at the point it successfully connects to a GC. Hopefully this blog will help explain the issue to your network team.

As mentioned above you can use Network Monitor to identify which Global Catalog servers the AD FS server is trying to connect to and then look at allowing Network Access to the first of the IP addresses you find.

2) Initialize Device Registration

Initializing Device Registration would create the DRS configuration containers in Active Directory and should avoid the Global Catalog discovery steps that lead to the delay.

Device Registration can be initialized using the following PowerShell script on one of the AD FS 2012 R2 servers:-

Initialize-ADDeviceRegistration

Note – this needs to be run with Enterprise Administrator credentials and requires that the Windows 2012 R2 AD schema update has been deployed. For further details on this cmdlet please see the following article - http://technet.microsoft.com/en-us/library/dn479332.aspx

This cmdlet does *not* enable the device authentication nor the device registration service in the ADFS servers. Enabling Device Registration is a second step in the DRS set-up process.

3) Tune the TCP SYN Retransmit behavior

If you are unable to implement either of the above resolutions then tuning the TCP SYN retransmit behavior can also be used to workaround the problem.

As discussed previously, we will incur a 21 second delay for each Global Catalog server we try to connect to that we cannot reach. If we can tune the timeout down to that the cumulative delay brings this under 100 seconds (let’s say 80 seconds to give some time for other required activities) then the AD FS server should respond to the configuration request within the 100 second timeout.

- The default setting for TcpMaxConnectRetransmissions is 2 and this is also the minimum value so we cannot tune this further down.

- This leaves us with tuning InitialRTO down from its default value of 3000 msecs. The valid range is 300-3000 msec. This can be tuned using the following netsh command:-

netsh interface tcp set global initialRto=3000

It’s hard to give a precise value to set this to as it how low you will need to go will depend on how many Global Catalog servers the AD FS server is trying to reach. You can use the following table to get an idea of the timeout per server you will get for various initialRto settings:-

initialRTO setting (in milliseconds)	Total Timeout Period (per failed GC connection)
3000 (default)	21 seconds
1000	7 seconds
500	3.5 seconds

This would only be a viable option for up to around 20 unreachable Global Catalog servers. Anything above this number is going to start to reach the 100 second overall timeout even with a 3.5 second total timeout per server.

Note – lowering the initialRTO could potentially cause the AD FS server connectivity issues in high latency environments and while we would not expect this to be the case for the majority of deployments please just keep this in mind. This is also a global OS level setting so affects any outbound TCP connections made by the AD FS server. It has no impact on inbound connections.

Note – reducing the initialRTO is not something that we would normally recommend so please do remember if you do change this that we’d recommend that you revert to the default settings when we have a fuller resolution for this issue if you have needed to tune this down.

Summary

Under certain conditions the Web Application Proxy post-installation configuration task may fail with the following error:-

“AD FS Proxy could not be configured. Unable to retrieve proxy configuration data from the Federation Server.”

The AD FS event log on the Web Application Proxy server will also show an Event ID 422 error with “System.Net.WebException: The operation has timed out” in the error details.

We do not expect the issue we’ve covered here to be a common issue but due to the difficulties in diagnosing this and understand the resolution options we wanted to share our findings.

In the cases we have seen this timeout errors due to the AD FS server trying to locate the Device Registration configuration containers in Active Directory. As part of this discovery the AD FS server may try to connect to GC’s across the local Forest.

If there is a complex, segregated network infrastructure in place these connection attempts may fail incurring a 21 second delay per inaccessible Global Catalog server leading to the Web Application proxy server 100 second timeout being exceeded.

There are a number of workarounds you can consider to alleviate this issue:-

- Allow ADFS connectivity to one of the relevant Global Catalog servers

- Initialize Device Registration

- Tune the TCP SYN retransmit timer (initialRTO) to reduce the time delay per GC server

We are working with the AD FS Product Group on a solution for this issue moving forward and will update this article when we have some updates on this but in the meantime hopefully the above information will help unblock any of you who do hit this issue.

Ian Parramore, Senior Escalation Engineer, Web Application Proxy support team

Thanks also to the following for helping put all the pieces of this jigsaw together:-

Billy Price, Senior Escalation Engineer, Web Application Proxy support team

Rainier Amara, Support Escalation Engineer, Web Application Proxy support team

As part of the bigger Windows announcement this week we are excited to share with you the abundance of additional functionality that has been added to Web Application Proxy and AD FS as part of Windows Server next version Technical Preview.

It has only been a year and a half since we exposed Web Application Proxy to the world in Windows Server 2012 R2 Preview. In this short time period it was deployed by many customers. Many Fortune 500 companies, Big Four accounting firms and also thousands of small companies and organizations from all over the world, have adopted Web Application Proxy.

We heard your praises that it is a robust architecture and the easiest reverse proxy to deploy. We tried to keep ourselves tuned to hear also the criticism and the requests for features you need. Over this year and a half we talked to dozens of customers that deployed the product and looked closely how we can help them get more value from their deployments.

We also talked to customers that are still using our legacy products, Forefront TMG and Forefront UAG that were announced as end-of-life and examined the barriers moving to Web Application Proxy.

At the end of this process we decided to focus on two themes:

1. Enable publishing and preauthentication for more apps
   
2. Ease the experience even more

We are eager to hear what you have to say about those new features. Please share with us your experience. It will help us improve the product towards the official release. We want to know how these features perform in your environment and if they satisfy your needs. You can use the Email Blog Author option on this page to contact us.

You can install the new Web Application Proxy first by downloading the Windows Server next version Technical Preview bits on a new machine or try it on Azure and then by adding the role service. We recommend installing it with Windows Server vNext AD FS, although some of the functionality will also work even when deploying it with Windows Serve 2012 R2 AD FS.

Our TechNet documentation has all the details on the new features. Here are the highlights:

HTTP Basic preauthentication – Secure ActiveSync traffic

In this release we are adding the capability to preauthenticate with AD FS protocols that are using HTTP Basic authentication where credentials are sent within the request and clients do not understand HTTP redirects or able to present login UI. For example, one of the most common protocols that is working like that is Exchange ActiveSync. With this addition, you can preauthenticate Exchange ActiveSync traffic with Web Applications Proxy.

When a request for a URL of application that is marked as HTTP Basic arrives, Web Application Proxy extracts the credentials from the request and sends this data to AD FS using a new AD FS endpoint. AD FS processes the request as it is doing for all authentication requests. If it is valid, it issues a token that is sent back. When Web Application Proxy receives a valid token from AD FS, it allows the traffic to be sent to the backend server. The token is cached by Web Application Proxy for future use to reduce the AD FS load.

This capability is on-top of the existing preauthentication methods that Web Application Proxy supports with AD FS that includes: Browsers, MSO-FBA for Office clients and OAuth2.

Here is a diagram describing this flow:

HTTP Basic preauthentication with certificates – Making ActiveSync even more secure

HTTP Basic will have an additional mode that would allow not only to preauthenticate the user but also to assure that the request is coming from a device that has gone through the process of workplace join and is registered using the device registration service (DRS). This is done by requiring incoming requests to present the workplace joined device certificate that was issued to them. This certificate is validated by AD FS using the same endpoint that is used to validate username and password.

This mode allows the ultimate security, making sure that the client presents proof of “what you know” (password) and “what you have” (device).

Wildcard domain publishing – new patterns and easier SharePoint 2013 apps publishing

We are adding the ability to publish not only a specific domain name but an entire sub-domain. This opens new opportunities for customer that want to publish sites in bulk and not one by one. For example, if all your apps are under http://*.apps.internal/, you can publish them using a single external domain like https://*.apps.contoso.com/.

This pattern is important when there is a need to publish SharePoint 2013 apps that uses a special sub-domain to all apps. In this case, only wildcard certificates would work as the specific apps domain may change over time.

HTTP publishing

In Windows Server 2012 R2, applications cannot be published with an external URL that is not HTTPS. This was done to assure that there is no leakage of user’s data. In current version we allow also using external address with HTTP.

HTTP publishing will be available only to applications that are not using preauthentication to assure that sensitive information will not leak.

HTTP to HTTPS redirection

We heard from many customers that when they publish applications using HTTPS, users should be able to access them even if they typed in their browser’s address bar the application HTTP URL with the same domain. By redirecting HTTP requests to the external URL of HTTPS applications we reduce the risk of server spoofing and improve the user experience.

In previous version, configuring this was complicated and required IIS installed side-by-side to Web Application Proxy. In the current version we are enabling this feature without installing IIS and with a single Boolean parameter or UI checkbox. Whenever this option is selected for HTTPS application, HTTP redirection will kick in.

Here is a sample configuration of both HTTP publishing and HTTP to HTTPS redirect:

Remote Desktop Gateway (RDG) publishing

Remote Desktop is a very common method to allow remote employees accessing legacy applications, usually rich applications that cannot be published using reverse proxies. In order to simplify the deployment experience for our customers, we have made a change in Web Application Proxy to enable publishing of Remote Desktop Gateway. This change allows RDG to pick up the session cookie that was used by RD Web Access so the RDP over HTTP traffic is authenticated.

This opens new opportunities as AD FS can be utilized to perform preauthentication for Remote Desktop access using all of its capabilities such as Multi-Factor Authentication and smartcards. For the users, it means that Remote Desktop authentication has the same experience as their Web applications. It also simplifies the admin experience by having a single entry point to the system and a single authentication and authorization mechanism across all applications.

This change was made available also to customers using Web Application Proxy on Windows Server 2012 R2 as part of the August 2014 update package.

Here is a typical topology of such deployment:

Improved logging for better audit and easier troubleshooting

Logging is important aspect of Web Application Proxy and in this release we wanted to make them more comprehensive. In order to do it without compromising performance, we added many more events as analytics and debug logs. Other than these new events, we also added more information to the existing admin events.

Administrator UI Enhancements

We LOVE PowerShell. But, we hear from many customers that they also love our admin UI and would like to use it for more scenarios. In this release we are improving the UI and provide more options that were PowerShell-only in Windows Server 2012 R2. We have added the ability to edit the applications directly in the UI.  This will allow administrators to review their configuration settings without having to go to PowerShell.

All of the new features described above are available on both our administrative console UI and using PowerShell. As usual, you can use the console to generate PowerShell commands and run them later.

Propagate client IP address to backend application

Many customers told us that for troubleshooting and auditing purposes, they want their backend applications to be aware of the original client IP address. To solve this, Web Application Proxy will add now to every request X-Forwarded-For header that includes this address. If this header already exist, we will concatenate the client IP to this header.

The list of new features is long. Our team worked hard on delivering the new capabilities while assuring we continue our track record of high quality and security standards. Now we are waiting to hear from you. Please let us know not only on things that do not work for you but also on the things that are working fine. We have long road ahead of us to bring this into the final release and maybe even to add more capabilities.

In parallel to evolving Web Application Proxy as traditional reverse proxy, we are investing in making Azure Active Directory Application Proxy an innovative service that would allow customer to move to new cloud patterns. Most of these capabilities will be available on both solutions enabling customers to select the type of deployment.

Hi everyone,

We’re pleased to introduce our new guest blogger, Mark Grimes from Microsoft Consultancy Services. Mark’s been involved in some pretty complex Web Application Proxy deployments and has put together some great hints and tips we wanted to share. Over to Mark…..

Introduction

In this posting I want to share with you some of the fundamental rules that you need to follow when publishing applications in Web Application Proxy. 

Perhaps a more accurate title for this might be, “paths that work and those that don't”?  No matter what the title, if you learn some simple rules to follow, you will improve your paths to success when publishing applications in Web Application Proxy. 

Web Application Proxy is a new role in Windows Server 2012 R2 that allows an administrator to publish access to an internal web application for external users. 

If you do not follow these paths to success, then you will often be politely asked to stop what you are doing right before you think you are done with your publishing. In this blog, we’ll show you some examples of those GUI stop signs, which will be presented to you as you attempt to do something that is simply not allowed and we’ll also explain the reasons why.  In some cases you may receive a warning just as soon as you enter your external and internal URLs. But sometimes you will just have to wait a few clicks before receive your surprise announcement declaring that you cannot do what you just thought you could do. And as administrators, we do not like surprises. 

Knowing these rules in advance will help you plan your Web Application Proxy deployment and avoid unforeseen issues when you’re right in the middle of your deployment. 

Let's summarize these "paths to success" to help you be successful in publishing applications through Web Application Proxy. Then we'll break them down one by one with examples for each.

1. External publishing through HTTPS only

2. Web Application Proxy can translate the host portion of the fqdn names but not path names

3. The absolute path must have a “/” suffix

4. Web Application Proxy supports wildcard certificates, but not wildcard host names for publishing

5. Nesting is for the birds, not Web Application Proxy

1. External publishing through HTTPS only

For publishing of web sites, port rules are simple. Externally, Web Application Proxy only currently only supports secure external publishing with HTTPS. This is to be secure by design.  Web Application Proxy supports connecting to the internally published server over either HTTP or HTTPS URLs. The only thing that will stop this from happening is if you do not follow the rest of the rules after this one.  Here is the example warning you will get as soon as you currently try to type HTTP for the external URL.

So you say, “Hey, what about all of those HTTP only sites we have that we need to publish?”

As you’ll no doubt have seen from Meir’s recent blog support for HTTP publishing along with a ton of other great new functionality is being added to Web Application Proxy in Windows Server vNext :-

http://blogs.technet.com/b/applicationproxyblog/archive/2014/10/01/introducing-the-next-version-of-web-application-proxy.aspx

In the interim, one example of an easy work around, which many of our customers do, is to implement something like an F5 load balancer iRule, which does a simple URL rewrite from HTTP for inbound requests to the network, to HTTPS traffic heading to Web Application Proxy.

2. Web Application Proxy can translate the host name portion of the published path but not path names

Web Application Proxy supports different internal and external FQDN’s e.g. https://www.contoso.com externally, but with an internal server name of http://secret-server.net. With an internal name like that, we surely do not want to share that name with the world. Note, in this example, the external URL is HTTPS and the internal URL is HTTP and it’s worth just noting this as although we currently only support external access over HTTPS we can bridge to HTTP internally. There is no path after the host name portion (which would have to match), so this will work and apply to all paths under the FQDN. Here is what it looks like:

Note: a warning dialog box does come up, but it is just noticing that the host names indeed are different. 
After clicking Publish in the Publishing wizard, you will get a successful confirmation.
 
To simplify the second part of this rule with an example, which is “Web Application Proxy cannot translate path names”, we’ll keep the external and internal host portions of the names the same but only change the path names following, to make them different.

In this first example, let’s try to publish https://contoso.com/app1/ (externally) to https://contoso.com/secret-app/ (internally).

You get the same warning as above, but you can still click through this and will not be blocked until the very end once published

To fix this, match the path following the FQDN host name portions for BOTH the external and internal names.

Following are successful examples with the same hostname and pathnames and then publishing apps with different host name portions of the fqdn, but the same path names following the hostnames.

If you have not done publishing with Web Application Proxy yet, the first scenario in the picture above is how the Web Application Proxy wizard assumes by default that you will publish the apps.  This means that as soon as you type in your application’s external name, it auto-fills in the application’s internal name to be the same.  This is by design and helps to fully mitigate any host translation issues.

3.    The absolute path must have a “/” suffix

This one is a quick and simple rule.  What this means, is to always end any application path with a trailing “/”. In the example below, https://contoso.com/app1 is attempted to be published (and not https://contoso.com/app1/). Once Publish was clicked, the interface complained once again with the dialog box below.

But here is a little twist even to this one! My customer had some paths that did not have a trailing slash. We’ll use the failed publishing attempt above as a requirement. Well no worries mate! If you publish https://contoso.com/app1/and the customer needs https://contoso.com/app1 (sans slash), this will still work as it is implicitly allowed by design.

4.    Web Application Proxy supports wildcard certificates, but not wildcard host names for publishing

There will be other blog posts covering certificates, but we just want to differentiate these two “wild” examples.
As shown below, wildcard certificates such as *.contoso.com are used to quickly publish any application without having to make a new certificate for every name. This means that for any host name that start with something.contoso.com, this certificate will work for just the one level beyond the dns name of contoso.com.  We’ll stop right there to save the rest of the certificate tips and tricks for another later blog post.

Next, we’ll try this using wildcard HOST names and see what happens.

The warnings in RED let us know that this is not valid to use wildcard HOST names.

This is an important requirement especially for publishing Apps for SharePoint where each application has a GUID FQDN under the app namespace e.g. contosoapps.com and is another feature that is added in the vNext release:-

http://blogs.technet.com/b/applicationproxyblog/archive/2014/10/01/introducing-the-next-version-of-web-application-proxy.aspx

To complete the set, if you are thinking about wildcards in the URL path, well this is just simply implied and allowed.  So if you publish https://contoso.com/ internally and externally. Then ANY sub-path beneath that will work as well e.g.

•    https://contoso.com/app1/secure/
•    https://contoso.com/app2/public/

…will ALL be allowed to pass through the gateway if you published https://contoso.com/ 

In one customer scenario, they have many sub paths for different apps or fedlets within their Java applications.  In this case things like:

•    https://contoso.com/app1/ is equivalent to https://contoso.com/app1/*
•    https://contoso.com/app2/secure/ is equivalent to https://contoso.com/app2/secure/*

5.    Nesting is for the birds, not Web Application Proxy

This final scenario we’re going to look at is what happens when you combine overlapping or nested path publishing.
Nesting paths in Web Application Proxy  is a situation where there is one path published initially, which is then later published later with the same initial fqdn followed by an overlapping path.  As an example, if you tried to publish the following internal URLS:

•    https://contoso.com/app1/
AND   
•    https://contoso.com/app1/public/

…here is the warning you will get.

Besides trying to avoid these “nestings”, how else can you avoid this?   One option would be to use alternate DNS host names.  If you published the following:

•    https://contoso.com/app1/
AND   
•    http://public.contoso.com/app1/public/

..this WILL work.  Why? Remember rule #2 above?  Web Application Proxy can translate host portion of the path but not pathnames between the internal and externally published URLs.

We do recognise that this is a requirement for a number of scenarios and it’s something the Product Group are evaluating for vNext.

Summary

There you have it. Five rules for “paths to success” in publishing applications in Web Application Proxy. If you follow these rules, may your publishing be successful ever after! And I do hope that these rules help you to avoid the common errors as shown above. Just like anything we do, practice makes perfect, or at least makes fewer mistakes.

Till the next time – Mark @ Microsoft

Hola,

TechEd Europe is around the corner and we have lots of stuff for you - Other than the dedicated session on Web Application Proxy and Azure AD Application Proxy (EM-B318) we are going to have a special announcement and various events. Some of it will be recorded so you can watch it at home. We will post links here when the event is over, so stay tuned!

More than anything, we would like to use this opportunity to meet you in person and hear about your deployments and the challenges you are facing with the application proxies. So if you are coming to Barcelona, drop us a line at aadapfeedback@microsoft.com so we know you are there. And of course, we’ll also be at the Tech Expo and on Ask The Expert evening.

See you there,

Meir Mendelovich & Arieh Bibliowicz

Hi everyone, this blog took a bit longer to get out than we’d planned but we hope it’s worth the wait.

A common question we’ve been seeing from various sources is how to configure Hardware Load Balancer health checks to work correctly with Web Application Proxy. There are some key considerations you need to be aware of when configuring your Load Balancer health checks, but if you know about these things, then it should be pretty straight-forward.

We will also look at new functionality introduced in the August 2014 Windows Server 2012 R2 Update rollup (KB.2975719) to provide HTTP Health Check endpoints for your AD FS 2012 R2 and Web Application Proxy servers.

Key Properties of a Health Check request

First, let’s look at the different properties at various protocol layers that we are interested in for a health check request and the recommended values.

While some of these are self-explanatory let’s drill into the details of what each of these mean and the considerations to take into account. The main ones we have seen some confusion and issues around are SNI Header and Host Header.

1) TCP level – the IP address and port the health check request will be sent to

Probably an obvious one but You will want your health requests to be sent to the dedicated IP address of each of the Web Application Proxy farm members.

2) TCP level – the port the health check request will be sent to
The health check requests should be sent to the relevant port that the service/application is listening on - most likely port 443 in a Web Application Proxy / AD FS scenario.

3) SSL level – does your load balancer support Server Name Indication (SNI) and if not what are your options?

If you are making a health check to an SSL endpoint, after the TCP connection has been established, the Load Balancer will send an SSL Client Hello to initiate the establishment of the SSL tunnel.

Web Application Proxy and AD FS 2012 R2 both use Server Name Indication (SNI) http.sys certificate bindings. SNI allows multiple certificates to be used on the same IP/port pair and http.sys uses the SNI header in the SSL Client Hello to determine which certificate should be served. This relies on the client sending the Server Name extension in the SSL Client Hello.

If the SSL Client Hello does not contain the SNI header then http.sys is unable to determine which certificate to serve and will reset the connection.

You therefore need to determine if your Load Balancer supports SNI in its health check requests. If it does support SNI then ensuring that the Load Balancer sends the SNI header in the SSL Client Hello is our recommendation.

Unfortunately there seems to be limited information available around configuring SNI with the common hardware load balancer solutions.

The following F5 article has some good details though especially around configuring the F5 Health Checks to send an SNI header - https://devcentral.f5.com/articles/big-ip-and-adfs-part-5-working-with-adfs-30-and-sni

Note– while this article has some great information on SNI and Health Check requests it also recommends using SSL Bridging which can break Web Application / AD FS traffic especially if bridging is used between Web Application Proxy and the AD FS 2012 R2 servers. Please see the following blog for more details on when and why SSL termination can cause issues:-

http://blogs.technet.com/b/applicationproxyblog/archive/2014/07/04/ssl-termination-with-web-application-proxy-and-ad-fs-2012-r2.aspx

For more details on SNI, how to detect if SNI headers are being sent and for options if your load balancer does not support SNI then please see the following blog article:-

http://blogs.technet.com/b/applicationproxyblog/archive/2014/06/19/how-to-support-non-sni-capable-clients-with-web-application-proxy-and-ad-fs-2012-r2.aspx

Using the 0.0.0.0 fallback binding is something we are seeing needed in a number of deployments although our recommendation would be to use SNI where you can.

4) HTTP Host Header – this should be the FQDN of the service being tested

This might seem quite an obvious one, but we have seen a number of issues where the Hardware Load Balancer is sending the dedicated IP address of the farm member as the Host Header. This can cause health check failures due to the following:-

- A Web Application Proxy published application will register the external application FQDN with http.sys and will not have a registration for the local dedicated IP address

- AD FS Proxy on the Web Application Proxy will not handle requests where the host header is the local dedicated IP address

- When SNI is used, http.sys will validate that the host header in the request matches the SNI header used during the SSL negotiation

It is therefore very important to ensure that your Load Balancer sends the correct service FQDN as the Host Header in the health check request e.g. if you are health checking the AD FS Proxy service you would send the ADFS Service FQDN as the host header e.g.

Host: adfs.contoso.com

For a Web Application Proxy published application you would set the host header to the External FQDN of the published application.

Not specifying the Host Header or, sending the node IP address as the host header will lead to unpredictable and/or failing health check behavior.

The F5 article mentioned in the SNI section above has the correct host header behavior so if you are using F5 load balancers it’s definitely the place to start. The following Cisco forum discussion talks about setting the Host header in a Cisco health check probe:

https://supportforums.cisco.com/discussion/10406021/setup-ace-probe-http-host-headers-multiple-sites-rservers

It’s important to note that when you are health checking either a Web Application Proxy published application or the AD FS Proxy service, that content is served from the respective back-end server. You are therefore health checking not just the availability of the local service, but also the communications to, and availability of, the relevant back-end service. A failure at the back-end could therefore potentially cause your load balancer to mark a Web Application Proxy node as offline even if other applications are available depending on your HLB configuration.

UPDATE -  as Jaydee points out in his comment if you are using pre-authentication for your Web Application Proxy published application(s) then you will get a 307 Redirect back when you health check your published application URL. This is expected and is Web Application Proxy redirecting to AD FS for authentication.

5) Which URL path to specify in the health check request

As mentioned above, neither Web Application Proxy, or the AD FS Proxy service serve content themselves and content will be retrieved from the back-end service.

For a Web Application Proxy published application, the path specified should be a relevant and valid path on the published application that will return a known response that the hardware load balancer can use for validation e.g. https://extapp.contoso.com/test.html

For health checks  against the AD FS Proxy service (or in fact AD FS itself) the current recommendation from the AD FS Product group is to health check against the following URL:-
https://adfsservicefqdn/FederationMetadata/2007-06/FederationMetadata.xml

Does it really have to be this hard?

While we’ve covered a lot of technical detail in the above the health check issues we’ve been seeing people hit generally come down to one or two of the above areas:

- Does the Hardware Load Balancer support SNI?
If it does then make sure the HLB is correctly configured to send an SNI header. If it does not then using the 0.0.0.0 fallback binding on your AD FS / Web Application Proxy server should provide a workaround.

- Ensuring that the correct Host Header is defined in the health check probe config
Host headers have been in use for many years as a way for a server to differentiate between requests for different sites. All HLB’s should be able to have a host header defined in their health check probes. If you are health checking against AD FS / AD FS Proxy then make sure you send the AD FS Service FQDN as the host header.

If you are aware of these 2 points and ensure these are taken into account when configuring your Hardware Load Balancer then getting your deployment up and running should be a lot smoother.

Disclaimer– while we have provided a couple of links out to HLB vendor sites the best people to advise you how to correctly configure your Hardware Load Balancer will be the relevant vendor. If you have any questions or doubts about how to correctly configure your Hardware Load Balancer please contact your vendor for advice.

No, I said “Does it really have to be this hard?”

Well, it is a good question even if you did have to ask it twice :-)

We’ve definitely heard the feedback that configuring Hardware Load Balancer probes for Web Application Proxy and AD FS 2012 R2 can be tricky. The issues generally boil down to a couple of key points and SNI and hardware load balancers particularly seems to fall into the realm of the mysterious and unknown.

We’re therefore pleased to be able to tell you about some new functionality that shipped in the August 2014 Windows Update rollup for Server 2012 R2 - http://support.microsoft.com/kb/2975719/en-us

*NEW* - HTTP based Health Check Probe functionality

The August 2014 Windows Update rollup (http://support.microsoft.com/kb/2975719/en-us) shipped new HTTP based health check probe functionality in ADFS 2012 R2 and the AD FS Proxy service that runs as part of the Web Application Proxy server role.

This avoids any issues with SSL certificates and SNI. It also supports the use of both FQDN’s and IP addresses in the probe helping avoid any potential host header issues. The new HTTP probe can be accessed over HTTP using the path /adfs/probe e.g.

http://<Web Application Proxy name>/adfs/probe

http://<ADFS server name>/adfs/probe

http://<Web Application Proxy IP address>/adfs/probe

http://<ADFS IP address>/adfs/probe

The response to these probe endpoints is an HTTP 200 OK and is server locally with no dependence on back-end services and allows the availability of the local AD FS and AD FS Proxy services to be checked. Health checks for Web Application Proxy published services would still work as detailed in the earlier part of this blog.

These probe endpoints are available by default but on a Web Application Proxy server you will need to create an Inbound Windows Firewall rule to allow the port 80 traffic in to the probe e.g.:

AD FS 2012 R2 already has an Inbound Port 80 rule so you do not need to create a manual Windows Firewall rule to use the HTTP probe against AD FS.

While resolving issues with health-checking over SSL due to SNI or host headers issues is relatively straight-forward, it does require that the issues are understood and an accompanying level of information about the associated Hardware Load Balancer configuration.

We hope that providing an HTTP based health check probe option gives an alternate solution that has less complexity to ease your deployments.

Summary

Hardware Load Balancer health check requests and AD FS 2012 R2 and Web Application Proxy should be fairly straight-forward but we have seen some challenges in a couple of specific areas:-

- Does the Hardware Load Balancer support SNI?

- Ensuring that the correct Host Header is defined in the health check probe config

Understanding these and taking them into account during your deployment is important to avoid health check request failures. If you are unsure about whether your load balancer supports SNI or, how to configure your load balancer to meet the above requirements, please do reach out to your HLB vendor. We’ll try to share what we can here but the best people to advise on how to configure your HLB correctly is your HLB vendor.

If you come across any good info on configuring any of the Hardware Load Balancer variants out there or links to relevant vendor configuration steps please feel free to share and comment below.

We have also added new functionality in the August 2014 Windows Update rollup for Server 2012 R2 (http://support.microsoft.com/kb/2975719/en-us) to provide HTTP based health check probes to provide another health check option, which avoids some of the complexities such as SNI and host header configuration.

We’ll look forward to any feedback, comments and questions you have and we’ll be back with more content in the near future.

Ian Parramore, Senior Escalation Engineer, Web Application Proxy support team

Hola from Barcelona!

This is a big day for Azure AD Application Proxy. We have just announced three new capabilities that were added to the service and available today for all of you to use:

1. Preauthentication with Azure AD

2. Authorization using users and groups assignments

3. Access panel integration

We also disclosed that we plan to make the service generally available and drop the preview tag before the year ends.

See Alex Simon blog post for more details.

Earlier today, in TechEd Europe keynote session Julia White demonstrated the Application Proxy along other new things coming from Azure Active Directory. Check it out here starting at 1:34.

You can also check out this Channel9 interview on the application proxy and some Azure AD background.

Tomorrow, we are going to do a deep dive session discussing and demoing all the new capabilities. We will let you know when the recording will be available.

Meir Mendelovich

Hi,

Two weeks ago we had TechEd Europe in Barcelona. It was an amazing event with exciting keynote, hundreds of breakout sessions and many one-on-one discussions. Though the event is over, lots of its content is available for you to watch on-line via our MSDN Channel 9 Web site. The key session on both application proxies was our "free your apps" session. You can watch it here and download its slides: http://channel9.msdn.com/Events/TechEd/Europe/2014/EM-B318

Here are few more sessions that I think would be interesting for you:

Windows Server Web Application Proxy

John Craddock teach you how to troubleshoot AD FS and Web Application Proxy: http://channel9.msdn.com/events/TechEd/Europe/2014/EM-B410

Samuel Devasahayam from the AD FS team explain on how to do conditional access:  http://channel9.msdn.com/events/TechEd/Europe/2014/EM-B310

Microsoft Multi-Factor Authentication (MFA/2FA) explained for on-prem (works with Web Application Proxy) and cloud (works with Azure AD Application Proxy): http://channel9.msdn.com/events/TechEd/Europe/2014/EM-B313

Azure AD Application Proxy

An informal interview for Channel 9 Live with yours truly on Azure AD and Azure AD Application Proxy: http://channel9.msdn.com/events/TechEd/Europe/2014/Ch9-06

Level 200 review on all Azure AD features and roadmap by Nasos Kladakis: http://channel9.msdn.com/events/TechEd/Europe/2014/CDP-B210 I can't resist quoting Nasos from this session: "With Application Proxy the sky is not the limit - you have on-prem"

Technical in depth explanations and demos of Azure AD Premium by Eran Dvir: http://channel9.msdn.com/events/TechEd/Europe/2014/CDP-B312

I'm sure that this would keep you busy,

Meir Mendelovich