Office 365 and Web Proxy – the Lost Documentation

BACKGROUND AND PURPOSE

Running Office 365 together with web proxy is supported and also the reality for many (or most) global Enterprise customers. Moving to Office 365 with web proxy present in your landscape is related to a number of considerations and possible configuration steps in order to ensure the best possible performance and functionality.

Guidelines for customers using web proxy do exist in the Office 365 Service Description and Deployment Guide. However, the information is scattered between different articles, why it can be unclear for the solution architect or network administrator to get a clear view on the considerations and principles that apply when using web proxy with Office 365. The purpose of this article is to collect the available information (plus some notes from the field) in one place to form an assembly on the given topic. Note that the information in this article also applies to CRM Online.

THE CLOUD AND WEB PROXY – WHAT COULD POSSIBLY GO WRONG?

Having a web proxy and running cloud based services might seem like the perfect fit. “The service is on the Internet, and our web proxy accelerates the Internet traffic to our end users web browsers – what could possibly go wrong?”.

There is no general answer to what could go wrong – it depends on what Office 365 services you plan to enable, and also what type of web proxy you are using.

But, to give an idea, here are six examples of things that might break when deploying Office 365 behind web proxy without first taking the right precautions:

  1.  Office Pro Plus Click-2-Run installation fails with the reason: “Something went wrong – Sorry, we ran into a problem”, as the installer changes from winhttp to BITSAdmin at around 17% of the installation progress. (Explained here).
  2. Outlook might lose its Connection to the Exchange server intermittently. A “balloon message” in the lower right explains “The connection to Microsoft Exchange is unavailable. Outlook must be online or connected to complete this action.”
  3. Downloading files from SharePoint Online takes an unreasonable amount of time to complete.
  4. Send and receive in Outlook works, except for downloading the Offline Address Book (gives error 0X8004010F). Sporadic credential prompts may also appear.
  5. Outlook might take several minutes to start/load. (Explained here).
  6. If using ADFS with Office 365, the Microsoft Identity Platform Relying Party Trust throws an error stating “This relying party is out of date due to monitoring issues. Please check the event log for details”.

To remediate all the example errors above, and many others like them, read through and act upon the following considerations:

CONSIDERATION #1 – IF POSSIBLE, BYPASS THE WEB PROXY COMPLETELY FOR OFFICE 365

If your network configuration and security policies allows you to bypass your web proxy for selected services/sites on the Internet, it is strongly recommended that you consider adding Office 365 to such exception list and open your client networks for direct access to Office 365s URLs (strongly recommended) (using IP-ranges does not work in practice and is not recommended as per this article). Generally, Office 365 will always work best without the client being behind a web proxy (reasons explained in the categories below). If making such exceptions is not an option in your case and you will use your web proxy with Office 365, keep reading the following considerations and recommendations.

RECOMMENDATION: If excluding services/sites from the web proxy is an option, and you can allow web traffic directly from the client networks to selected URLs on the Internet, use this article to find the specifications for what URLs are used by Office 365 and allow the traffic directly  (using IP-ranges does not work in practice and is  not recommended). The list is subject to updates, monitor the Change Notification RSS feed to keep your access lists current. Also remember to add all URLs to the proxy bypass list on the client (in your PAC file or in the Internet Explorer settings) to avoid Office 365 traffic to go through the web proxy spitefully). The list of additional URLs needed for CRM Online is found in this article.

For the majority of the services (like using the portal, SharePoint Online, OneDrive for Business, Exchange Online and the Outlook client), you will only need to open port 80/443, but additional ports are needed for Skype for Business. Specifications on ports and protocols are found here.

CONSIDERATION #2 – NAT:ED IP ADDRESSES AND NUMBER OF CONCURRENT SESSIONS

This topic applies even without web-proxy, but definitely relates to your web proxy configuration.

In the Office 365 Service Description, there is a topic called “Plan for Internet bandwidth usage for Office 365”, found here. In the planning guidelines, a topic can be found called NAT support with Office 365.

In that topic, Microsoft explains the limitations that applies when using Network Address Translation (NAT) in respect to the amount of ports that are available to share between users behind the NAT:ed IP-address (public/Internet facing address). I recommend reading the article for details (sub-topic “NAT support with Office 365”), but here is the summary:

Clients for Office 365 in general, and Outlook in particular, opens sessions with the service in the cloud. These sessions consumes the shared amount of ports that are available per Internet facing NATed IP-address. The general guidance is to add a public facing IP address per 2000 concurrent users to share. Personally I recommend adding one NAT address per 5000 concurrent users.

RECOMMENDATION: Read the article above and scale out the amount of public IP-addresses to provide for your number of concurrent users. For example, if you are using one proxy server with one single NATed address to the Internet to provide for 10 000 concurrent users, add one or two more IP-addresses to you web proxy configuration (configure a NAT pool, add NIC or add additional IP addresses, whichever works for your web proxy). Add more IP-addresses for more concurrent users.

CONSIDERATION #3 – RFC1323 (TCP WINDOW SCALING)

An excellent article on this topic can be found here: http://blogs.technet.com/b/onthewire/archive/2014/03/28/ensuring-your-office-365-network-connection-isn-t-throttled-by-your-proxy.aspx

The article explains how disabling the TCP Window Scaling mechanism in any network device may cause very bad performance to Office 365 (and probably other services as well). In the example used in the article, the download time for a 14 MB PDF file is reduced from over 8 minutes to just 32 seconds by simply enabling the Auto Scaling mechanism in the web proxy. That is pretty remarkable.

RECOMMENDATION: Make sure that RFC1323/TCP Windows Scaling is enabled in all network segments from your clients to the Internet. See the TechNet-article for details. If you are running Bluecoat, you can follow this article to find out if you have RFC1323 enabled or not: https://kb.bluecoat.com/index?page=content&id=FAQ1006

CONSIDERATION #4 – NECESSARY EXCEPTIONS (BEWARE OF CACHING, TRAFFIC INSPECTION AND AUTHENTICATION PROMPTS)

On of the key benefits with having a web proxy is its ability to cache content from the web to increase performance and decrease use of Internet bandwidth. However, the Office 365 services is not always fully compatible with content caching, and it is not recommended to use such technology. Read this post for official statement and details: http://support.microsoft.com/kb/2690045/

The article is addressing WAN Optimizers, but the principal applies to web proxy as well. The bottom line is that caching the data might interrupt the service protocols and thus deteriorate performance rather than improving it.

Content caching will not improve any performance in Outlook as the user is working against a local copy of the mailbox (the OST-file created when Outlook is running in Cached mode). There is often a debate if content caching can improve SharePoint Online performance when working with large files – but my recommendation is to try without content caching and later enable it just for SharePoint Online to compare. If there is no, or a minimal, performance increase – you best stay without content caching to comply with the principles stated in the article above. For Skype for Business Online, content caching can not help increasing performance either.

Further, web proxies have the ability to inspect web traffic for malicious content and filter out suspicious traffic before it can be downloaded to any client. These technologies go under many names (Content Inspection, DPI, DCI etc), but the principal is the same – traffic is inspected in transport and rules may filter it if found suspicious. Such content inspection might also disturb Office 365 client protocols, and cause intermittent disconnection and/or performance decrease.

Another element of the web proxy is its ability to secure web traffic by only allowing authenticated users to access the Internet. This will require authentication of some sort for the user. If single Sign-on (SSO) authentication is enabled for the web proxy, this should not be a problem with Office 365, but if the web proxy presents credential prompts (operating system based or as forms in the actual browser) as part of the day-to-day user experience, you can expect those prompts to be a an issue for Office 365 to function as well – traffic should be allowed without authentication requirements.

RECOMMENDATION: Three parts are covered in this consideration and recommendation – content caching, traffic inspection and authentication without seamless SSO. None of these components are fully compatible with Office 365, why you should make sure to exclude Office 365 URLs in the web proxy for all of them. The web proxy will function just as a proxy/relay for the Internet traffic, but will not cache its content, inspect its traffic or require authentication if the user have not already authenticated. The URLs to exclude can be found in this article. After adding the exception list in your web proxy, make sure to visit the Change log for URLs now and then to check for updates. There is also an RSS feed of the updates that you can subscribe to. The list of additional URLs needed for CRM Online is found in this article.

CONSIDERATION #5 – PAC SCRIPTS AND IE SETTINGS

There are different methods of distributing the web proxy settings to the clients in the network. One is to Proxy Auto-Config (PAC) files that you either distribute to your browser automatically with the WPAD protocol (DHCP and DNS lookups) or you can specify the URL to the PAC manually or by using Group Policies (just make sure to never use file:// paths). If not using PAC at all, you can specify the hostname directly to the web proxy, also manually or via Group Policy.

All methods works with Office 365, but problems may occur if you are not using PAC files, but your Internet Explorer settings suggest that you do (“Automatically detect settings” or “Use automatic configuration script” is enabled).

The first symptom of this is usually that Outlook takes several minutes to load (explained here). Another symptom might be that 3rd party applications that attempt to access Office 365 services (such as a mailbox in Exchange Online), simply can’t connect to the service. The Internet Explorer settings are very sensitive, and they might require some adjustments before everything operates as expected. Often, the setting “Bypass proxy server for local addresses” plays an important role in the trouble shooting.

RECOMMENDATION: Generally, the most solid setup for Office 365 together with web proxy is when using PAC files with the correct settings in Internet Explorer. If you are not using PAC files, make sure that your IE settings do not suggest that such file exist on the network, and keep your eyes and ears open for bad Outlook performance and similar indications.

SUMMARY AND DISCLAIMER

As stated initially, the first recommendation is to exclude Office 365 from the web proxy completely and allow clients to reach the specific URLs directly from the client. If such exception is not an option, the other considerations and recommendations in this article will hopefully help you to achieve the best possible performance and experience with Office 365 (and/or CRM Online).

Please note that this article is written based on field experience, and that the scenarios explained might or might not apply to your scenario and your specific web proxy. Any feedback to this article is very welcome to my Twitter, @JesperStahle. My intention is to keep this article up to date with guidelines for using web proxy with Office 365 – so if you know more considerations and recommendations that you should make the list, please let me know.

Hands on with Office 365 Message Encryption (OME) using One Time Passcode

ARTICLE UPDATED OCTOBER 2014 AFTER LAUNCH OF OTP CAPABILITIES IN OME.

INTRODUCTION

At the beginning of 2014, Microsoft released new capabilities for email encryption, called Office 365 Message Encryption (OME). The functionality was first announced in november 2013 in this article. In October 2014, OME was improved drastically as the receiver of encrypted emails now can use a One Time Passcode (OTP) to access the encrypted message (documented here). At launch of OME, all receivers had to authenticate with either an Organizational Account (Azure AD) or a Microsoft Account (formerly LiveID) to access the email, which often led to confusion for external receivers. Now, the receiver can instead request a One Time Passcode to the same email address as to which the encrypted message was sent, and use that (within 15 minutes) to access the encrypted email. This post was updated in October 2014 to include this new OTP-functionality.

In this post, I will demonstrate the user experience of Office 365 Message Encryption, both for the end-user (using OTP) and the administrator (setting up IRM Licensing and all necessary Transport Rules etc).

SCENARIO

We will create a transport rule that will enable Office 365 Message Encryption on messages with a Sensitivity level set to Confidential.

STEP 1 – ENABLE IRM LICENSING

If you attempt to use Office 365 Message Encryption before first enabling IRM licensing, the operation will fail and give you this message:

You can’t create a rule containing the ApplyOME or RemoveOME action because IRM licensing is disabled.

To remediate this, we need to enable IRM licensing using the Admin portal and PowerShell. Follow these steps:

  1. First, enable RMS in your tenant by logging on to your Office 365 Admin Portal, navigate to Service settings (left pane). select Rights management (top bar), click Manage and finally hit the button Activate. A message should appear stating that RMS is activated for the tenant.
  2. The remaining steps will be done in PowerShell. Open Windows Azure Active Directory Module for Windows PowerShell, found here:
    http://technet.microsoft.com/library/jj151815.aspx
  3. Connect your PowerShell session to your Office 365 tenant an Exchange Online by entering the following:
    $msolcred = Get-Credential (Answer the credential prompt with your Office 365 tenant administrator credentials) 
    Connect-MsolService -Credential $msolcred

    $Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $msolcred -Authentication Basic -AllowRedirection
    Import-PSSession $Session
  4. When connected to Exchange Online, you can enable IRM licensing with just a few steps. The first step is to set the RMS Online key sharing location. You will use different configurations depending on where your tenant is located (North America, European Union or the Asia-Pacific area).Enter the command that matches your tenant location (choose one):
    North America: Set-IRMConfiguration -RMSOnlineKeySharingLocation https://sp-rms.na.aadrm.com/TenantManagement/ServicePartner.svc
    European Union: Set-IRMConfiguration -RMSOnlineKeySharingLocation https://sp-rms.eu.aadrm.com/TenantManagement/ServicePartner.svc
    The Asia-Pacific Area: Set-IRMConfiguration -RMSOnlineKeySharingLocation https://sp-rms.ap.aadrm.com/TenantManagement/ServicePartner.svc
  5. After setting the key sharing location, the next step is to import the Trusted Publishing Domain (TPD). Do so by entering the following:
    Import-RMSTrustedPublishingDomain -RMSOnline -name “RMS Online”
  6. The final step is to activate the internal IRM licensing. Do so by entering the following:
    Set-IRMConfiguration -InternalLicensingEnabled $True
  7. After enabling IRM licensing, verify the functionality by entering:
    Test-IRMConfiguration -RMSOnline

The test should pass with the result OVERALL RESULT: PASS.

NOTE: Enabling IRM licensing might need several hours for the change to take effect. If possible, allow 8 hours to pass before proceeding with STEP 2.

STEP 2 – CREATE THE OME TRANSPORT RULE

We will create a transport rule that enables Office 365 Message Encryption if the message is sent to a recipient outside the organization and the Sensitivity header have been set to Confidential.

Follow these steps:

  1. Log in to your Office 365 Admin Portal and navigate to Exchange Control Panel (Admin\Exchange).
  2. Navigate to Mail Flow, click the + icon and select Create a new rule…
    Newrule
  3. Give the rule a suiting name and click More options…
    moreoptions
  4. From here you can set your condition as it fits your needs, but for this example we will inspect the Sensitivity header and apply Message Encryption based on that. To do so, select the following conditions:
    Apply this rule if… A message header includes any of these words
    amessageheader
  5. Complete the Apply this rule if-condition by clicking the properties Enter text and Enter word so that the condition makes ‘Sensitivity’ header includes ‘Confidential’
    sensincl
  6. Click Add condition and select The recipient… Is external/internal. Click Select one… and select Outside the organization and hit OK
    outside
  7. Proceed with clicking Do the following… and select Modify the message security… and select Apply Office 365 Message Encryption
    applyo365m
  8. Hit Save at the bottom of the New rule editor. (If you get the message that IRM licensing is not enabled and have completed STEP 1 – please allow more time for the change to take effect, as stated in the Note after STEP 1).

STEP 3 – SEND A CONFIDENTIAL MESSAGE WITH OME

We will create a message with the sensitivity level set to Confidential and send this to a recipient outside our organization. Our transport rule will apply Office 365 Message Encryption to the message.

Follow these steps:

  1. Open Outlook or Outlook Web App. Both clients have the native functionality to set the sensitivity header. In the example, I will use OWA, but the procedure is the same in Outlook.
  2. Compose a new message. Enter an external recipients, give the message a subject and some content, then click  and Show message options…
    messageopt
  3. Set the message Sensitivity level to Confidential and hit OK
    confidential
    NOTE: In the rich Outlook client, Sensitivity options are found under Message Options\More options\Sensitivity.
  4. Send the message.

STEP 4 – RECEIVE AND OPEN THE ENCRYPTED MESSAGE

  1. Open the inbox of the external mailbox and find the encrypted message
    protectedmessage
  2. As you can see, the message includes an attached HTML-document. Open the attachment.
  3. To be able to read the encrypted message, you can choose to sign in with a Microsoft Account or Corporate Credentials (AAD/Office 365 account) that has a username that matches the e-mail address to which the encrypted message was sent – or (new October 2014) you can choose to open the message with a One Time Passcode that is sent to the same email address that received the encrypted message (html file). I recommend using Organizational Accounts when sending to other Office 365 users, as they can then open the message with their corporate credentials or even an existing session cookie (Single Sign-on experience) – but for external users, which we are demonstrating in this case, One Time Passcode would be prefered. I am selecting the OTP choise by following the link at the bottom (“Don’t want to sign in? Get a one-time passcode…”).
    Select "Get a one-time passcode..."
  4. Now, go back to your inbox (recipient) to find the One Time Passcode and a reference code (handy if you have received multiple encrypted messages at once to differentiate the OTPs for different messages). Make a note of the OTP and go back to the browser.
    Find the OTP in your inbox
  5. Enter the Passcode from the email in the Passcode field, then click Continue.
    step2
  6. The message will now load and be displayed in the browser.
    voila

CONCLUSION

The Office 365 Message Encryption is a very good feature that allows Office 365/Exchange Online customers to send encrypted messages both inside and outside their organization. The new OTP functionality makes it even better and more complete. This post demonstrated the basic capabilities of OME, and you can make even more advanced Mail Rules, such as applying RMS templates to the message if it is sent to an internal recipient, and use Office 365 Message Encryption only to external recipients – just configure the rule to fit your needs.

You can also customize/brand the message that includes the HTML-file by using the
Set-OMEConfiguration cmdlets. The cmdlets allow you to set custom e-mail text, disclaimer, portal text and even include your company logo for branding.

 

Can’t login to Lync Online on Domain Joined Machine in a Single Labeled Domain (SLD)

BACKGROUND

  1. You are using Lync Online together with Lync or Lync Basic client.
  2. You’r  client is joined to an Active Directory domain with Single Label Name (SLD). FQDN Name is, as an example, “domain” and not “domain.local”
  3. You can logon to Lync using mobile clients or from other clients than are not domain joined to your Single Labeled Domain.
  4. When logging on with the Single Labeled Domain joined machine, the Lync Client gives the following error:
    The Server is temporarily unavailable. If the problem continues, please contact your support team.

CAUSE

Single Labeled Domains are not supported with Lync Server, but there is no documentation regarding Lync Online instead of an on-premise Lync installation in combination with a client running from a Single Labeled Domain.

The fact is that this combination does not work wihtout additional configuration (see Resolution). Allthough the Lync Online infrastructure is not running in a Single Labeled Domain, the client can’t login from a machine joined to a Single Labeled Domain.

RESOLUTION

To solve this, all users that are running Lync Online must have a proper UPN-suffix in Active Directory. There is NO NEED to login with the UPN-name, but the attribute itself must be populated.

Follow these steps to verify the solution with a few users, before applying to all users:

  1. In you Active Directory forest, add an additional UPN Suffix. As suggestions to what to add, you can add the same as your SMTP domain(s), or just use the current SLD and add “.com”. Instructions on how to add UPN Suffix can be found here: http://support.microsoft.com/kb/243629
  2. On your own (an/or other) user account(s), change the UPN Suffix to the newly added “domain.com-like” using Active Directory Users and Computers. You find the UPN suffix on the Account tab, the right drop-down box under “User logon name”
  3. Wait for AD Replication to complete (15 minutes give or take, depending on number of AD sites etc.)
  4. Log off and on to your workstation
  5. Attempt to log in to Lync again
  6. Repeat for remaining users

With a correct UPN Suffix, Lync Online will allow you to log on from the domain joined machine, regardless if you logged in with your samAccountName or the full UPN-name.

Shared Accounts (POS/Kiosk) and Single Sign-On in the Same Environment

BACKGROUND

You have successfully configured Single Sign-on in Office 365 with the following conditions:

  1. ADFS is deployed with one internal pair (STS) and one external pair (Proxy), both pairs are load balanced. ADFS is running at least version 2.0 with the latest Update Rollup installed.
  2. Clients that resides on the internal network have the ADFS URL (in this example sts.Company.com) pointing to the ADFS STS server pair (the LB VIP).
  3. Clients that resides on the external network have the ADFS URL (sts.Company.com) pointing to the ADFS Proxy pair (the LB VIP).
  4. On all Active Directory-domain joined clients, the ADFS URL (sts.Company.com) is added to the Local Intranet Zone using a Site-to-Zone Assignment List GPO.
  5. You have not configured ADFS to allow Single Sign-on for non-IE browsers (Disable Extended Protection).
  6. You have verified that Single Sign-on works as expected by navigating to http://outlook.com/OWA/company.com (where Company.com is a custom domain that is configured as a Federated Domain in your Office 365 tenant). When navigating to the URL, you are redirected to Outlook Web App without being prompted for credentials.

This scenario is exactly what we want for our users that are logging on to Windows everyday using their personal Active Directory account. It is a seamless solution that allow the user to consume Office 365 services without entering credentials multiple times through-out the working day.

The problem is if parts of the organization is using common/shared Active Directory accounts for logging on to Windows. A few examples of when common Active Directory accounts are used for logging on to Windows are at: Point of Sale terminals (POS), front desk/receptions, and at heavy industry operation sites (factory users are restricted by gloves/outfit).

These users will not be able to access their personal Office 365 services as they are logged in to Windows with the shared account, and when attempting to access, for example, Outlook Web App, they either get a message that the current account does not have a valid license (if the shared account is not licensed), or they will be logged in to the wrong mailbox (if a licensed user is logged on to Windows when the other user is taking over the session). The user have no chance to login as themselves to reach their personal mailbox etc, as we have configured Single Sign-on to Office 365, without any credential prompts allowed.

In summary, we have a situation where users logging on with personal Active Director accounts to their personal computers have a perfect experience with Single Sign-on, but our users without personal computes that are logging on to Windows with shared accounts, are simply not able to reach their mailbox and other data without first logging in to Windows with their personal account – which is not feasible for the business operation.

CAUSE

This bevaviour is fully expected. It is not ideal for users to share Active Directory accounts for loggin on to Windows, both from the security perspective (complicance and auditing) and from the functional perspective (single sign-on and personal data access). To remediate it, the general recommendation is of course to have all users logging on to Windows with personal Active Directory accounts, but in reality, this is not always possible. The reasons are many, and they are not described on TechNet or in any text book examples, but they are usually related to the time it actually takes to log off and log back on. At a POS, every lost second can be a lost sale, and at the factory floor, every lost second can be a compromise of a persons safety.

Technically, our Single Sign-on configuration is doing exactly what it is supposed to do. A user attempts to reach Outlook Web App, get’s redirected to ADFS STS and authenticates using the Kerberos TGT-ticket issued for the shared account when logging on to Windows. ADFS grants a Token, including claims for the shared account. The Token is then presented to Office 365, who translates the claims to either a non-licensed user (giving the No license error), or if it exist, to the mailbox for the shared account (loading the wrong mailbox for the user).

RESOLUTION

To remediate this problem, without affecting the Single Sign-on experience for our Office workers with personal computers, we must break the Single Sign-on configuration only for the shared accounts.

To accomplish this, we will configure the Site to Zone Assignment List GPO.

As stated in the Background above, the ADFS URL sts.Company.com is added to the Local Intranet Zone to allow Single Sign-on. If we remove it from there, the user will not be able to present the Kerberos TGT-ticket when logging on, and we will prompted for credentials again. This is a bad user experience and should never be the routine for Office users with personal computer and personal accounts, but such credential prompt is what can allow a user that has logged on with a shared user account to actually login to Outlook Web App (or other Office 365 Service) with a personal account and reach the personal mailbox.

High-level description of the solution:

  1. Depending on how Group Policy is used in your organization today, first decide if this setting should be computer based or user based. Both can be applied for this.
  2. Configure separate Site to Zone Assignment List GPO’s for personal Active Directory accounts and for common/shared Active Directory accounts. (Or for personal computers or shared computers, depending of outcome in #1).
  3. For the personal account/computer GPO, place sts.Company.com in the Local Intranet Zone – this will give them Single Sign-on
  4. For the shared accounts/computers, Place sts.Company.com in the Trusted Sites zone – this will break Single Sign-on and give the user a credential prompt on access to the services. The user can respond to the credential prompt with personal credentials and by that accessing the personal mailbox/data.

Here is a good overview of the Site to Zone Assignment List GPO: http://blogs.msdn.com/b/askie/archive/2012/06/05/how-to-configure-internet-explorer-security-zone-sites-using-group-polices.aspx (Scroll down a bit).

COMMENT: Some organizations solves this issue by letting all clients redirect sts.company.com to the ADFS Proxy even internally. This methdo will also present the credential prompt, but it will affect all users – including those that are using personal accounts on personal computers – which is a bad user experience for them. Using GPO and Site to Zone assignment is  a better way to go.

PST Capture Tool 2.0 Only Imports Deleted Items Folder

BACKGROUND

You are attempting to import data to Exchange Online mailboxes using the PST Capture Tool 2.0 (http://www.microsoft.com/en-us/download/details.aspx?id=36789). The following works:

1. You can import PST files to the tool
2. You can map the PST files to mailboxes in Exchange Online (x64 bit version of Outlook is required to see the mailboxes)
3. If you open the PST file in Outlook on the machine, you can see all the content

When starting the Import, the stage called “Transferring” completes 100%. Then, when the Import stage begins, you can see that it only finds 2 folders. The progress shows Importing Folder 0/2, 1/2 and then 2/2.

After the import is done, you can see in the Exchange Online mailbox that the only content that have been imported is the Deleted Items folder.

CAUSE

This occurs if the PST file that is imported is in an incompatible format. Without going into, or knowing, the details of it all, PST files can be created in different structures. The version of Outlook together with how PST Capture Tool 2.0 works decide if all content in the PST files root folder (IPM_SUBTREE) will be visible to the operation or not. Read more here: http://msdn.microsoft.com/en-us/library/ff387699(v=office.12).aspx

The error can occur when the PST file have been exported from Exchange Server 2010 SP3 using New-MailboxExportRequest.

RESOLUTION

Run SCANPST.EXE from the Program Files\Microsoft Office\(VersionNumber) folder on the machine. Use SCANPST.EXE to scan and repair the PST files you want to import. After repairing the file, PST Capture 2.0 will import all folders and objects.

This process takes a while, and if you are to do 100+ mailboxes, it might have a severe impact to your planning. Depending on the PST file, it might help to change Outlook version on the machine where you run PST Capture 2.0 – just make sure that it is always x64 bit.

Autodiscover and rich Outlook configuration fails but SSO for OWA/Lync/Portal works (“The AD FS 2.0 Windows Service service failed to start due to the following error: The service did not respond to the start or control request in a timely fashion.” ExRCA: “A network error occurred while communicating with the remote host. “) – CRL check in ADFS is failing

BACKGROUND

The following works:

  • You have successfully deployed ADFS and Single Sign-on  with Office 365
  • You can successfully log on to the Office 365 Portal, Outlook Web App and the rich Lync client using SSO (Active Directory credentials) both from the inside and outside (through ADFS Proxy)
  • You have added the correct CNAME for the Autodiscover service in the public and internal DNS zone for you custom domain
  • You can successfully access the ADFS metadata from the outside by opening https://sts.domain.com/adfs/services/trust/mex in the web browser – it displays content. (sts.domain.com is the URL to your ADFS Proxy)

The following does not work:

  • When attempting to configure an ActiveSync device with Autodiscover (entering e-mail address and AD-password), the configuration fails and the device asks for server name.
  • When attempting to configure your rich Outlook client, the configuration fails and repeatedly ask for username and password during Autodiscover configuration
  • ExRCA (http://testexchangeconnectivity.com) tests for Autodiscover fails with the following error:
    Testing TCP port 443 on host autodiscover.domain.com to ensure it’s listening and open.
      The specified port is either blocked, not listening, or not producing the expected response
    Additional Details
      A network error occurred while communicating with the remote host.

CAUSE

As SSO with the Office 365 portal, Outlook Web App and the rich Lync Client works, as well as accessing the ADFS Metadata, you will probably not think that this issue is ADFS related, but rather related to a back-end error of sort in the Exchange Online service. However, looking in to the ADFS configuration, you may find event 7000 with the following explanation:

The AD FS 2.0 Windows Service service failed to start due to the following error: The service did not respond to the start or control request in a timely fashion.

This is related to why Autodiscover, ActiveSync and the rich Outlook client configuration will not work.

If you investigate the network traffic while attempting to start the ADFS Service, you might find that the service is attempting to do a CRL check for the certificates. This fails, which causes the service start to time out – leaving you with an ADFS service that works good for Office 365 portal, OWA and Lync – but not Autodiscover, ActiveSync and rich Outlook.

This may occur if:

  1. The ADFS servers can’t access the Internet for CRL lookups
  2. You are attempting to use web Proxy for the CRL lookups, but the Proxy configuration is faulty (bad IE Connections settings or missing connectivity between ADFS servers and the web Proxy itself)

The failing CRL check will not reveal itself in the event log, you can however see it by monitoring the network traffic.

RESOLUTION

First, verify that the ADFS Service account have Log on as a Service and Log on as a batch job rights. (The behavior can occur if this is missing as well). Then, follow one of the solutions below to remediate the problem with CRL verification access from the ADFS servers.

Solution 1 – Fix the CRL Access:

  1. Verify that you have proper web Proxy (IE Connections) settings on the ADFS servers. Verify that you can access the Internet, or needed Internet hosts for CRL checks in your Environment.
  2. Start the service and see that it comes online properly.

Solution 2 – Turn off Automatic Root Certificates Update:

  1. Open gpedit.msc (Start\Run)
  2. Navigate to Computer Configuration \ Administrative Templates \ System \ Internet Communication \ Internet Communication
  3. Open Turn off Automatic Root Certificates Update and set it to Enabled
  4. Start the service and see that it comes online properly.

Note: Solution 2 is here described for the local Group Policy which applies if you are using Security Baseline configuration on the server. If an Active Directory GPO is controlling this setting, you need to set this in the AD GPO.

 

After applying Solution 1 or 2 above, and verifying the Log on as a service and Log on as a batch job rights for the ADFS Service Account, Autodiscover and rich Outlook configuration will work.

DirSync installation and/or operation fails Invalid Namespace (System.Management.ManagementException: Invalid namespace / just-in-time (JIT) debugging)

BACKGROUND

  1. You install DirSync and the installation fails with the error message Invalid Namespace. If you view additional information, you can see this just-in-time (JIT) debugging information: System.Management.ManagementException: Invalid namespace
  2. Or, you have successfully installed DirSync, but after a short while of operation, the same error appears in the event log and DirSync stops working.
  3. Or, you attempt to uninstall DirSync, and the same error appears.

CAUSE

This is caused by an error in the Windows Management Framework 3.0 that (long story short) causes the WMI repository of the server to be rebuilt. When the repository is rebuilt, data for the MicrosoftIdentityIntegrationServer (miis) in the WMI Control Properties are cleared.

The issue occurs if you have, or have had, an SCCM agent on the server that is on the RTM version, in combination with Windows Management Framework version 3.0.

You can read the full details of the error here: http://support.microsoft.com/kb/2796086

RESOLUTION

  1. To fix this issue, set the following Registry value to True:
    HKLM\SOFTWARE\Microsoft\CCM\CcmEval\NotifyOnly
  2. Re-install DirSync

Be aware that this issue may have a bigger impact in your environment than just causing DirSync and it’s installation to fail.  If you encounter this issue, make sure to read this KB-article: http://support.microsoft.com/kb/2796086 and fully resolve the error on all servers/clients by deploying System Center 2012 Configuration Manager SP1.

Free/Busy not working from Office 365/Exchange Online to On-Premise Exchange / Hybrid Configuration – “The external recipient’s server could not be determined” “No information” (You forgot to iisreset)

BACKGROUND

You have successfully deployed a Hybrid Configuration between your on-premises Exchange 2010 SP3 Organization and Exchange Online. Everything works as it should, except for Free/busy look-ups made from Exchange Online mailboxes to a mailbox that still resides on-premise.

You have verified that the following is OK:

  1. You have run ExRCA diagnostics using https://www.testexchangeconnectivity.com/ , selected the Office 365 tab and then Free/busy tests. You simulate Free/busy lookup from Online to On-premise and it works with all green lights.
  2. You have also followed all the troubleshooting steps presented in this Microsoft Knowledge Base article: http://support.microsoft.com/kb/2555008 – all with successfull results.

Still, when attempting to do Free/busy lookups from Online to On-premise, you get the message in Outlook or OWA saying “No information” with the reason “The external recipient’s server could not be determined”.

CAUSE

Settings made during your Hybrid Configuration and/or extended troubleshooting of this error have not been fully commited to Internet Information Server (IIS). Allthough the diagnostics (ExRCA etc) say that your configuration is right, vital configuration is not yet effective.

RESOLUTION

First step in resolving Free/busy errors is to follow the two steps mentioned in the Introduction to this post (ExRCA diagnostics and following KB2555008 – see above). If that does not work:

Doing iisreset on your CAS server publishing EWS/Hybrid could fix this issue.

Plan for a very short outage in Exchange services (OWA/EWS etc) and open the command prompt on your Hybrid CAS server. Then run the command iisreset

After doing IISReset, Free/busy lookups from Online to On-prem should start working within minutes.

I have seen this in the field where we spent days troubleshooting this issue on Exchange 2010SP3 hybrid configurations after finally solving it with this basic, but classic, method – restarting the services. There could of course be other reasons behind the error (i.e related to how Autodiscover is published to the internet), iisreset should not be considered as a “general cure”, but a necessary troubleshooting step if your configuration is done by the book and it still does not work.

“Enable Centralized Mail Transport” / “Route all Internet-bound messages through your on-premises Exchange servers” and NDR Remote Server returned ‘550 5.7.1 Unable to relay’

BACKGROUND

You configure the Hybrid with the option to route back e-mail from Exchange Online users to your On-premise organization Before sending them to the Internet.

  • In the Exchange 2013 Hybrid Configuration Wizard, this option is called: “Enable Centralized Mail Transport”
  • In the Exchange 2010 Hybrid Configuration Wizard, this option is called: “Route all Internet-bound messages through your on-premises Exchange servers”

This is not the default option, but is necessary for some organizations due to compliance or message tracking reasons.

What it does is that it is adding a wildcard (*) in the send connector in FOPE/EOP, that normally only is used for your shared domains, to route back e-mail to users On-premise that have not been migrated to Exchange Online. The wildcard enables this send connector for all domains on the Internet, thus route the e-mail back to your On-premise Exchange Environment.

Your On-premise Exchange Environment can without any problem route e-mail to your internal, not yet migrated users, because that Exchange Environment knows about those domains and their recipients. Internet recipients, on the other hand, will require your message from the Exchange Online user to be RELAYED outside your On-premise Exchange, back to the Internet.

This fails with the following NDR (bounce message) information to the sender in Exchange Online:

Remote Server returned ‘550 5.7.1 Unable to relay’

CAUSE

The receive connector that the Hybrid Configuration Wizard creates on-premise to Catch the e-mail that Exchange Online sends back is not fully configured and ready for use after running the Hybrid Wizard.

RESOLUTION

To fix the “Inbound from Office 365” receive connector, you need to both populate the list of allowed IP-addresses and enable the receive connector for relaying.

Follow these steps:

  1. Open the Receive Connector properties in EMC or ECP (2010 or 2013 on-premise)
  2. First, open the Network tab/settings to investigate the list of remote servers, “Receive mail from remote servers that have these IP addresses”.
  3. The list of allowed IP-addresses is supposed to be populated by the Hybrid Configuration Wizard – BUT IT IS NOT COMPLETE! Use this article to verify each address space is included in this list: http://help.outlook.com/en-us/140/gg263350.aspx . Make sure to read the complete article to understand that the list of IP-addresses is subject to change etc.
  4. After making sure that your list of allowed IP-addresses is complete, you still need to enable the receive connector to allow further relaying to Internet recipients when the message origins from Exchange Online:
  5. Fire up Exchange Management Shell and run this PowerShell command to enable the “Inbound from Office 365” receive connector for relaying:

Get-ReceiveConnector ” Inbound from Office 365 ” | Add-ADPermission -User “NT AUTHORITY\ANONYMOUS LOGON” -ExtendedRights “Ms-Exch-SMTP-Accept-Any-Recipient”

After filling up the list of allowed IP-addresses and enabled it for relaying to Internet recipients, your mail flow will work as expected with the “Centralized Mail Transport” / “Route all Internet-bound messages through On-premise” option enabled.

Note: Be careful not to add excessive IP-address ranges to your receive connector. You do not want to create an open relay – make sure that you only allow Exchange Online hosts to relay.

 

 

This mailbox exceeded the maximum number of large items that were specified for this request. (Fatal error TooManyLargeItemsPermanentException has occurred.)

POST UPDATED TO REFLECT THE EXCHANGE ONLINE ONBOARDING SIZE LIMIT INCREASE (FROM 35 to 150 MB) ANNOUNCED JANUARY 2015 (INFO).

BACKGROUND

When moving mailboxes to Exchange Online, the Remote Move Request fails with the following error message in Exchange Management Console (EMC):

Error: This mailbox exceeded the maximum number of large items that were specified for this request.

When viewing the log file for the mailbox migration, you see the following details:

Fatal error TooManyLargeItemsPermanentException has occurred.

CAUSE

The message size limits for Exchange Online migrations is 150 MB. It used to be 35 MB prior to January 2015, but have now been increased (info). If larger items than 150 MB are attempted to be moved through a default Remote Move Request initiated from the EMC wizard, the job will fail with the error message above because Exchange Online refuses the messages due to their size. If you have mailboxes containing messages that are +150 MB in size, you can still migrate these mailboxes but you will need to tell Exchange that it should simply skip the large messages that are refused and not abort the complete move request when encountering them.

RESOLUTION

If a mailbox has items that are 150 MB+ in size, you can solve this error in two ways:

  1. Either instruct the end-user to remove the item. Send an instruction on how to sort items on size in Outlook and tell them to Shift+Del Everything above 150 MB in size. Elite users can maybe even handle an instruction on Search Folders/Filter. After the users have removed the items, remove the Move request and start over again (resuming it will fail).
  2. Or, allow the move request to skip the 150 MB+ items that are refused by Exchange Online and continue the migration without aborting the entire job. Here is how:

Unfortunately, you cannot use the EMC wizard for this, you will need to use the Exchange Online Control Panel (where the migration wizard will allow you to adjust the parameters) to run the migration or use PowerShell. Using the Exchange Online Control Panel works for both Exchange 2010 and Exchange 2013 hybrid configurations.

2.1: USING THE EXCHANGE ONLINE CONTROL PANEL INSTEAD OF EMC (RECOMMENDED)

The most simple way to solve this problem is to run the migration from the Exchange Control Panel (as said, this works with both Exchange 2010 and 2013 hybrid configurations). Follow these steps:

  1. Log in to you Office 365 Admin Portal and navigate to Exchange Control Panel (Admin\Exchange).
  2. Navigate to Recipients (left pane) and then the tab Migration (top pane/tabs)
  3. Click + (plus) and Migrate to Exchange Online
  4. Select the first choice to initiate a remote mailbox move
  5. Select users to migrate or use a CSV to create a new migration batch
  6. At the next step, enter your MRS endpoint, that is where EWS is published (your hybrid server). Example: ews.company.com
  7. At the next screen, you can give the batch a name for tracking and logging purposes, select if you want to migrate an Archive Mailbox, and most importantly set the allowed number of large and corrupt/bad items to be skipped for the migration. Increase this number to be able to run the migration that failed. As an example, you can set both values to 50.
  8. At the next screen, enter who will have reports when the migration batch is completed, set scheduling options (start now or later) and select if the migration should be completed directly or if it should be automatically suspended when the mailbox replication have reached 95%, so that you can switch over the user when you want in a controlled manner. If using the auto suspend option, the user will not notice anything until you choose to complete the migration, then the last 5%/delta will be synchronized and the user will be switched over (email will be forwarded to Exchange Online mailbox and Outlook will re-configure for use with the Exchange Online mailbox etc).
  9. When completing the wizard, you can then track the migration from the Migration view in the control panel. If you are using automatic suspend, here is also where you can complete the migration after reaching 95% (mark the job and select complete).

2.2: USING POWERSHELL

As stated above, using Exchange Online Control Panel is recommended. If you would like to use PowerShell instead, follow this method.

If you are moving just one user, follow these steps:

New-MoveRequest -remote -Id firstname.lastname@smtpdomain.com -RemoteHostName “ews.companydomain.com” -RemoteCredential $Cred -TargetDeliveryDomain ‘tenantdomain.mail.onmicrosoft.com’ -SuspendWhenReadyToComplete -LargeItemLimit 50 -BadItemLimit 50

Note that I added LargeItemLimit and BadItemLimit. These parameters will allow up to 50 large (over 150 MB) messages to be refused and skipped, and up to 50 corrupt messages to be skipped as well.

If you are moving several users, you could use a CSV file with a column that says UPN, filled with your migration batch users UPN-names:

Import-Csv C:\MigrationBatch.csv | foreach {New-MoveRequest -Identity $_.UPN -Remote -RemoteHostName ews.companydomain.com -TargetDeliveryDomain ‘tenantdomain.mail.onmicrosoft.com’-BadItemLimit 50 -LargeItemLimit 50}

For both single and bulk, consider adding the following parameter to use the AutoSuspend feature:

 -SuspendWhenReadyToComplete -Suspend

Moving mailboxes with EMC is very convenient, but it will not give you the option to allow corrupted or large (150 MB+) items to be skipped and overseen in the migration job. Go Exchange Online Control Panel or PowerShell to save the day.