BACKGROUND AND PURPOSE
Running Office 365 together with web proxy is supported and also the reality for many (or most) global Enterprise customers. Moving to Office 365 with web proxy present in your landscape is related to a number of considerations and possible configuration steps in order to ensure the best possible performance and functionality.
Guidelines for customers using web proxy do exist in the Office 365 Service Description and Deployment Guide. However, the information is scattered between different articles, why it can be unclear for the solution architect or network administrator to get a clear view on the considerations and principles that apply when using web proxy with Office 365. The purpose of this article is to collect the available information (plus some notes from the field) in one place to form an assembly on the given topic. Note that the information in this article also applies to CRM Online.
THE CLOUD AND WEB PROXY – WHAT COULD POSSIBLY GO WRONG?
Having a web proxy and running cloud based services might seem like the perfect fit. “The service is on the Internet, and our web proxy accelerates the Internet traffic to our end users web browsers – what could possibly go wrong?”.
There is no general answer to what could go wrong – it depends on what Office 365 services you plan to enable, and also what type of web proxy you are using.
But, to give an idea, here are six examples of things that might break when deploying Office 365 behind web proxy without first taking the right precautions:
- Office Pro Plus Click-2-Run installation fails with the reason: “Something went wrong – Sorry, we ran into a problem”, as the installer changes from winhttp to BITSAdmin at around 17% of the installation progress. (Explained here).
- Outlook might lose its Connection to the Exchange server intermittently. A “balloon message” in the lower right explains “The connection to Microsoft Exchange is unavailable. Outlook must be online or connected to complete this action.”
- Downloading files from SharePoint Online takes an unreasonable amount of time to complete.
- Send and receive in Outlook works, except for downloading the Offline Address Book (gives error 0X8004010F). Sporadic credential prompts may also appear.
- Outlook might take several minutes to start/load. (Explained here).
- If using ADFS with Office 365, the Microsoft Identity Platform Relying Party Trust throws an error stating “This relying party is out of date due to monitoring issues. Please check the event log for details”.
To remediate all the example errors above, and many others like them, read through and act upon the following considerations:
CONSIDERATION #1 – IF POSSIBLE, BYPASS THE WEB PROXY COMPLETELY FOR OFFICE 365
If your network configuration and security policies allows you to bypass your web proxy for selected services/sites on the Internet, it is strongly recommended that you consider adding Office 365 to such exception list and open your client networks for direct access to Office 365s URLs (strongly recommended) (using IP-ranges does not work in practice and is not recommended as per this article). Generally, Office 365 will always work best without the client being behind a web proxy (reasons explained in the categories below). If making such exceptions is not an option in your case and you will use your web proxy with Office 365, keep reading the following considerations and recommendations.
RECOMMENDATION: If excluding services/sites from the web proxy is an option, and you can allow web traffic directly from the client networks to selected URLs on the Internet, use this article to find the specifications for what URLs are used by Office 365 and allow the traffic directly (using IP-ranges does not work in practice and is not recommended). The list is subject to updates, monitor the Change Notification RSS feed to keep your access lists current. Also remember to add all URLs to the proxy bypass list on the client (in your PAC file or in the Internet Explorer settings) to avoid Office 365 traffic to go through the web proxy spitefully). The list of additional URLs needed for CRM Online is found in this article.
For the majority of the services (like using the portal, SharePoint Online, OneDrive for Business, Exchange Online and the Outlook client), you will only need to open port 80/443, but additional ports are needed for Skype for Business. Specifications on ports and protocols are found here.
CONSIDERATION #2 – NAT:ED IP ADDRESSES AND NUMBER OF CONCURRENT SESSIONS
This topic applies even without web-proxy, but definitely relates to your web proxy configuration.
In the Office 365 Service Description, there is a topic called “Plan for Internet bandwidth usage for Office 365”, found here. In the planning guidelines, a topic can be found called NAT support with Office 365.
In that topic, Microsoft explains the limitations that applies when using Network Address Translation (NAT) in respect to the amount of ports that are available to share between users behind the NAT:ed IP-address (public/Internet facing address). I recommend reading the article for details (sub-topic “NAT support with Office 365”), but here is the summary:
Clients for Office 365 in general, and Outlook in particular, opens sessions with the service in the cloud. These sessions consumes the shared amount of ports that are available per Internet facing NATed IP-address. The general guidance is to add a public facing IP address per 2000 concurrent users to share. Personally I recommend adding one NAT address per 5000 concurrent users.
RECOMMENDATION: Read the article above and scale out the amount of public IP-addresses to provide for your number of concurrent users. For example, if you are using one proxy server with one single NATed address to the Internet to provide for 10 000 concurrent users, add one or two more IP-addresses to you web proxy configuration (configure a NAT pool, add NIC or add additional IP addresses, whichever works for your web proxy). Add more IP-addresses for more concurrent users.
CONSIDERATION #3 – RFC1323 (TCP WINDOW SCALING)
An excellent article on this topic can be found here: http://blogs.technet.com/b/onthewire/archive/2014/03/28/ensuring-your-office-365-network-connection-isn-t-throttled-by-your-proxy.aspx
The article explains how disabling the TCP Window Scaling mechanism in any network device may cause very bad performance to Office 365 (and probably other services as well). In the example used in the article, the download time for a 14 MB PDF file is reduced from over 8 minutes to just 32 seconds by simply enabling the Auto Scaling mechanism in the web proxy. That is pretty remarkable.
RECOMMENDATION: Make sure that RFC1323/TCP Windows Scaling is enabled in all network segments from your clients to the Internet. See the TechNet-article for details. If you are running Bluecoat, you can follow this article to find out if you have RFC1323 enabled or not: https://kb.bluecoat.com/index?page=content&id=FAQ1006
CONSIDERATION #4 – NECESSARY EXCEPTIONS (BEWARE OF CACHING, TRAFFIC INSPECTION AND AUTHENTICATION PROMPTS)
On of the key benefits with having a web proxy is its ability to cache content from the web to increase performance and decrease use of Internet bandwidth. However, the Office 365 services is not always fully compatible with content caching, and it is not recommended to use such technology. Read this post for official statement and details: http://support.microsoft.com/kb/2690045/
The article is addressing WAN Optimizers, but the principal applies to web proxy as well. The bottom line is that caching the data might interrupt the service protocols and thus deteriorate performance rather than improving it.
Content caching will not improve any performance in Outlook as the user is working against a local copy of the mailbox (the OST-file created when Outlook is running in Cached mode). There is often a debate if content caching can improve SharePoint Online performance when working with large files – but my recommendation is to try without content caching and later enable it just for SharePoint Online to compare. If there is no, or a minimal, performance increase – you best stay without content caching to comply with the principles stated in the article above. For Skype for Business Online, content caching can not help increasing performance either.
Further, web proxies have the ability to inspect web traffic for malicious content and filter out suspicious traffic before it can be downloaded to any client. These technologies go under many names (Content Inspection, DPI, DCI etc), but the principal is the same – traffic is inspected in transport and rules may filter it if found suspicious. Such content inspection might also disturb Office 365 client protocols, and cause intermittent disconnection and/or performance decrease.
Another element of the web proxy is its ability to secure web traffic by only allowing authenticated users to access the Internet. This will require authentication of some sort for the user. If single Sign-on (SSO) authentication is enabled for the web proxy, this should not be a problem with Office 365, but if the web proxy presents credential prompts (operating system based or as forms in the actual browser) as part of the day-to-day user experience, you can expect those prompts to be a an issue for Office 365 to function as well – traffic should be allowed without authentication requirements.
RECOMMENDATION: Three parts are covered in this consideration and recommendation – content caching, traffic inspection and authentication without seamless SSO. None of these components are fully compatible with Office 365, why you should make sure to exclude Office 365 URLs in the web proxy for all of them. The web proxy will function just as a proxy/relay for the Internet traffic, but will not cache its content, inspect its traffic or require authentication if the user have not already authenticated. The URLs to exclude can be found in this article. After adding the exception list in your web proxy, make sure to visit the Change log for URLs now and then to check for updates. There is also an RSS feed of the updates that you can subscribe to. The list of additional URLs needed for CRM Online is found in this article.
CONSIDERATION #5 – PAC SCRIPTS AND IE SETTINGS
There are different methods of distributing the web proxy settings to the clients in the network. One is to Proxy Auto-Config (PAC) files that you either distribute to your browser automatically with the WPAD protocol (DHCP and DNS lookups) or you can specify the URL to the PAC manually or by using Group Policies (just make sure to never use file:// paths). If not using PAC at all, you can specify the hostname directly to the web proxy, also manually or via Group Policy.
All methods works with Office 365, but problems may occur if you are not using PAC files, but your Internet Explorer settings suggest that you do (“Automatically detect settings” or “Use automatic configuration script” is enabled).
The first symptom of this is usually that Outlook takes several minutes to load (explained here). Another symptom might be that 3rd party applications that attempt to access Office 365 services (such as a mailbox in Exchange Online), simply can’t connect to the service. The Internet Explorer settings are very sensitive, and they might require some adjustments before everything operates as expected. Often, the setting “Bypass proxy server for local addresses” plays an important role in the trouble shooting.
RECOMMENDATION: Generally, the most solid setup for Office 365 together with web proxy is when using PAC files with the correct settings in Internet Explorer. If you are not using PAC files, make sure that your IE settings do not suggest that such file exist on the network, and keep your eyes and ears open for bad Outlook performance and similar indications.
SUMMARY AND DISCLAIMER
As stated initially, the first recommendation is to exclude Office 365 from the web proxy completely and allow clients to reach the specific URLs directly from the client. If such exception is not an option, the other considerations and recommendations in this article will hopefully help you to achieve the best possible performance and experience with Office 365 (and/or CRM Online).
Please note that this article is written based on field experience, and that the scenarios explained might or might not apply to your scenario and your specific web proxy. Any feedback to this article is very welcome to my Twitter, @JesperStahle. My intention is to keep this article up to date with guidelines for using web proxy with Office 365 – so if you know more considerations and recommendations that you should make the list, please let me know.