Hello folks,

The topic for today's post was borne out of the number of queries we've received from customers asking about configuring the Azure AD app proxy connector to work with existing proxies on-premises. In many organizations, network policy does not allow internal machines to access the internet directly. Therefore, going through an on-premises proxy to connect to the Azure service is a requirement to make the scenario work.

This is a setup that many of you may have to grapple with. To that end, Ian Parramore on our team has been leading the effort to examine what it takes to make the connector traffic work in customer environments with proxies. And he’s written up a very detailed set of steps to consider.

Hope this unblocks you and speeds up your adoption of Azure AD App proxy. As always, if you have any questions or feedback, please contact us at aadapfeedback@microsoft.com.

 BTW, am about to hop on a plane. I'm on my way to Hong Kong for the Azure Cloud Roadshow where I'll be talking about Azure AD and all its cool capabilities. If any of you are there, do drop by and say hi!

Thanks,

Girish Chander. 

Hi everyone,

 I’m Ian Parramore, one of the Senior Program Managers in the Azure Active Directory Go-To-Production team. We’ve been working with various customers to configure the Application Proxy connector to work with outbound proxy servers. In this article we’ll look at 2 main deployment scenarios:

 

-        Configuring Connectors to bypass your on-premise outbound proxies

-        Configuring Connectors to use an outbound proxy to access Azure AD Application Proxy

 

We’ll look at each of these scenarios in details, call out things to watch out for, as well as some troubleshooting tips and tricks.

 

But first some background:

• The Connector install contains two services:
  • Microsoft AAD Application Proxy Connector—this is the core Connector service that establishes the connection with the proxy
  • Microsoft AAD Application Proxy Connector Updater—this is the updater service that is responsible for checking for and applying updates to the Connector service.

Both these services will need to be configured explicitly

• The Core connector service uses Azure Service Bus to handle some of the underlying communications to the Azure AD Application Proxy service. Service Bus brings with it additional configuration requirements.
• If you have an outbound proxy in your environment, ensure that the account doing the install is appropriately configured to use the outbound proxy. As the installer runs in the context of the user doing the install this can be done in Internet Explorer, Settings, Internet Options, Connections, LAN Settings and configuring the necessary proxy settings.

 

Please change ‘proxyserver’ and ‘8080’ to reflect your local proxy server name and the port it is listening on.

 

Bypassing outbound proxies

By default, the underlying OS components used by the Connector for making outbound requests automatically attempt to locate a proxy server on the network using WPAD (Web Proxy Auto-Discovery), if it is enabled in the environment.

This typically works by carrying out a DNS lookup for wpad.domainsuffix. If this resolves in DNS, an HTTP request will then be made to the IP address for wpad.dat which will be the proxy configuration script in your environment. The Connector will then use this to select an outbound proxy server.

However, connector traffic may still not be going through because of additional configuration needed on the proxy.

 

In the next section we’ll cover the configuration steps needed on the outbound proxy to make the traffic flow through it. But first, let’s cover how you can configure the connector to bypass your on-premises proxy to ensure it uses direct connectivity to the Azure services. This is the recommended way to go (as long as your network policy allows for it) as it means that you have one less configuration to maintain.

 

To disable outbound proxy usage for the connector, edit the C:\Program Files\Microsoft AAD App Proxy Connector\ApplicationProxyConnectorService.exe.config file and add the following section in bold:-

 

To ensure that the Connector updater service also bypasses the proxy, make a similar change to the ApplicationProxyConnectorUpdaterService.exe.config file located at C:\Program Files\Microsoft AAD App Proxy Connector Updater\ApplicationProxyConnectorUpdaterService.exe.config

Please ensure you take copies of the original files in the event that you need to revert to the default .config files. 

 

Using the outbound proxy server

As mentioned above, in some customer environments, there is a requirement for all outbound traffic to go through an outbound proxy without exception. As a result, bypassing the proxy is not an option.

You can configure the connector traffic to go through the outbound proxy as shown below.

 

There are two steps to perform:

1)     Configure the connector and Updater service to go through the outbound proxy

2)     Configure the proxy settings to permit connections to the Azure AD App proxy service

 

 Step 1: Configure the connector and related services to go through the outbound proxy

As covered above, if WPAD is enabled in the environment and configured appropriately, the connector will automatically discover the outbound proxy server and attempt to use it. However, you can explicitly configure the connector to go through an outbound proxy.

To do so, edit the C:\Program Files\Microsoft AAD App Proxy Connector\ApplicationProxyConnectorService.exe.config file and add the following section in bold.

 

 

Step 2: Configure the Proxy to allow traffic from the Connector and related services to flow through

There are 4 aspects to consider at the outbound proxy

A.     Proxy Outbound Rules

B.     Proxy Authentication

C.     Proxy Ports

D.     SSL Inspection

 

2.A: Proxy outbound Rules

Allow access to the following endpoints for Connector service access.

• *.msappproxy.net
• *.servicebus.microsoft.net

Additionally, for initial registration, allow access to the following end points

• login.windows.net
• login.microsoftonline.com.

 

The underlying Service Bus control channels used by the connector service additionally require connectivity to specific IP addresses. (We are working with the Service Bus team to move to a fully FQDN instead). For now however, there are two options: 

1)     Allow the connector outbound access to all destinations

2)     OR, Allow the connector outbound access to the Azure Datacenter IP Ranges listed here https://www.microsoft.com/en-gb/download/details.aspx?id=41653

Note – the challenge with using the Azure Datacenter IP Ranges list is that this gets updated weekly so you would need to put a process in place to ensure your access rules are updated accordingly.

 

2.B: Proxy Authentication

Proxy authentication is not currently supported. Our current recommendation is to allow the Connector anonymous access to the Internet destinations.

 

2.C: Proxy ports

The Connector makes outbound SSL based connections using the CONNECT method. This essentially sets up a tunnel through the outbound proxy. Some proxy servers by default only allow outbound tunneling to standard SSL ports such as 443. If this is the case, the proxy server will need to be configured to allow tunneling to additional ports.

Configure the proxy server to allow tunneling to non-standard SSL ports – 8080, 9090, 9091 and 10100-10120.

 Special service bus considerations

When Service Bus runs over HTTPS it uses port 443. However, by default, Service Bus will attempt direct TCP connections and will fall back to HTTPS only if direct connectivity fails.

To ensure that the Service Bus traffic is also sent through the outbound proxy server you need to ensure that the Connector cannot directly connect to the Azure services on ports 9350, 9352 and 5671.

 

2.D: SSL Inspection

Do not use SSL Inspection for the Connector traffic as it will cause issues for the Connector traffic.

  

And that’s it, now you should see all traffic flowing through the proxy. If you run into issues, the following troubleshooting steps should help.

 

Troubleshooting Connector Proxy problems and service connectivity issues

 

The single best way of identifying and troubleshooting Connector connectivity issues is to take a Network capture on the Connector service while starting the connector service. This can seem quite daunting so let’s look at quick tips on capturing and filtering Network traces.

 

You can use the tool of your choice such as Network Monitor, Wireshark or Message Analyzer. For the purposes of this blog I’m going to use Network Monitor 3.4 which you can download here –  https://www.microsoft.com/en-us/download/details.aspx?id=4865

 

Examples and filters will be specific to Network Monitor but the principles can be applied in any analysis tool.

 

Taking a Capture using Network Monitor

• To start a Network Monitor capture select New Capture and then the Start button
  

•  After taking a capture, use the Stop button to stop the capture.

 

Taking a capture of Connector traffic:

For initial troubleshooting perform the following steps: -

 

1.      From the services.msc stop the Microsoft AAD Application Proxy Connector service:-

 

 

2.      Start the Network capture

3.      Start the Microsoft AAD Application Proxy Connector service

4.      Stop the Network capture

 

Looking at the Connector to Proxy server requests

 

Now that you’ve got a Network Capture we can move on to the fun parts

The key to looking at the trace is understanding how to filter the capture to find what we’re interested in.

 One of my favourite filter for this is as follows (where 8080 is the proxy service port):

 (http.Request or http.Response) and tcp.port==8080

 If you enter this filter in the Display Filter window and select Apply it will filter the captured traffic based on the filter.

 

The above filter will show just the HTTP requests and responses to/from the proxy port. For a Connector start-up where the Connector is configured to use a proxy server this would show something like this:-

 

We are specifically looking for the CONNECT requests which show we are talking to the proxy server and then that we get an HTTP OK (200) response back.

              

If you see other response codes such as 407 or 502 this would indicate that the proxy is requiring authentication or not allowing the traffic for some other reason at which point you should engage your proxy server support team.

  

Identifying failed TCP Connection attempts

              

The other common scenario we are interested in is when the Connector is trying to connect directly but this is failing.

 

Another one of my favorite Network Monitor filters which helps us easily identify this is:-

 

                              property.TCPSynRetransmit

 

Without going into too much detail a SYN packet is the first packet sent to establish a TCP connection. If this doesn’t get a response, then the SYN will be re-tried. The above filter allows us to see any re-transmitted SYN’s and we can then look to see if these correspond to any Connector related traffic.

 

The following example shows a failing connection attempt to the Service Bus port 9352

 

 

If you see something like the image above, it means firstly that the Connector is trying to talk directly to the Azure Service Bus service. If you expect the Connector to be making direct connections to the Azure services, then this is a clear indication of a network/firewall issue.

 

As noted above if you are configured to use a proxy server this may just be Service Bus attempting a direct TCP connection before switching to try connecting over HTTPS so please keep this in mind.

 

 

Network trace analysis is not everyone’s cup of tea but can be a hugely valuable tool to get some quick information about what is going on. If you are struggling with Connector connectivity issues though, please do feel free to raise a ticket with our support team who would be happy to assist with further troubleshooting.

 

Thanks,

Ian Parramore

Senior Program Manager, Azure Identity and Security