Tag Archives: Office 365 Grid

ADFS 3.0 and Hardware Load Balancing – the Lost Documentation

Article updated October 2014 to include http based health check probe as a workaround, made available with August 2014 Update Rollup for Windows Server 2012 R2.

BACKGROUND AND PURPOSE

ADFS and Federated Identities are common components for Enterprise Office 365 customers necessary to allow Single Sign-on, use of Claim Rules and immediate account lock-outs etc. As the availability the ADFS service decides the availability of Office 365 (if you can’t authenticate you can’t use the service), load balancing is a must-have. In most cases, Hardware Load Balancing (HLB) is used, and for ADFS 2.0 (Windows Server 2008/2008R2) and ADFS 2.1 (Windows Server 2012) this have never been a problem. You deployed your ADFS STS and/or ADFS Proxy, and published them with your Hardware Load Balancer like “any other SSL site” and it just worked.

Today, ADFS have with Windows Server 2012R2 reached version ADFS 3.0 – and the full “out-of-the-box-support” for using “any Hardware Load Balancer” is not as obvious any more. Making it  work is a bit more tricky, depending on what HLB you are using. The purpose of this article is to present my lessons learned from deploying ADFS 3.0 with HLB – and to save you from hours of trouble shooting when getting your own ADFS 3.0 HLB up and running with Office 365.

ADFS 3.0 AND HARDWARE LOAD BALANCING – WHAT COULD POSSIBLY GO WRONG?

ADFS 3.0 and Web Application Proxy (WAP) in Windows Server 2012R2 uses an extension to the TLS SSL protocol called Server Name Indication – SNI (RFC 4366). This extension allows web servers to present host names when handshaking SSL, so that multiple SSL sites can be hosted on a shared IP-address and port (443) – just like the concept of host headers in good old http have always allowed us to do. This is great because it allows preservation of IP-addresses which also can reduce a total cost of ownership for a service (public IP-addresses does not come for free these days).

What is not so great is that the SNI extension is not yet supported by “everything else in the world” – such as very old web browsers – and Hardware Load Balancers. Using a HLB that does not have a clue about SNI will not work, and that is the topic of this blog post.

If you are interested to read more about SNI in Windows Server 2012R2 – and how to solve issues that can occur, I can recommend Ian Parramore’s two excellent blog post on TechNet, which also have been a great inspiration and source of content to this post, found here, and here.

HOW DO I KNOW THAT MY HLB DOES NOT SUPPORT SNI?

If your HLB does not support SNI, you will notice it eventually when attempting to configure a load balancing and you don’t get any response on port 443 from any of the server IP-addresses. (that’s usually when you spend a day or two troubleshooting firewall ACL’s and routing without any luck). My best advice is to look for SNI-related settings in the HLB administrative interface and read through the release notes for the software version you are running on the appliance(s).

Examples of observed behaviour when attempting to configure HLB with a non-SNI compliant device is a “Http/1.1 Service Unavailable” message from a Citrix NetScaler device and a “Network Error (tcp_error) A communication error occured: Connection refused” message from a BlueCoat device.

Big-IP/F5 customers may turn to a technical article written by Greg Coward for assistance in configuring the Big-IP to work with ADFS 3.0/WAP, found here.

MY HLB DOES NOT SUPPORT SNI – WHAT CAN I DO?

CHECK FOR UPDATES: Before attempting to fix the issue on the Windows Server side – you should always see if there is an available software/firmware update for your HLB device that includes SNI support. There are two workarounds available (see below), but having native support is usually preferred as it reduces the overall complexity of your solution.

WORKAROUND #1 : ADD A HTTPS (PORT 80) BASED HEALTH CHECK PROBE

NOTE: This functionality was made available with the August 2014 Update Rollup for Windows Server 2012R2, found here. Running at least this update rollup level on your ADFS/WAP servers is required for the following workaround to work.

By adding a http-binding on the ADFS/WAP server, your HLB can do the health check probing using the http protocol and thus bypassing SSL and SNI completely. The http endpoint will respond with HTTP 200 OK if available if queried to http://servername-or-ip-address/adfs/probe, and will not automatically allow your ADFS traffic to use HTTP, they will function just as health check probes. Follow these steps to create such binding:

  1. Make sure you have installed all available Windows Updates on the ADFS/WAP machine – at least August 2014 Rollup.
  2. Launch the Windows Firewall with Advanced Security MMC on the first WAP server
  3. Go to Inbound Rules
  4. Create a new Inbound Rule (New Rule in Action Pane). Suggested name: “ADFS HTTP Health Check Probe”
  5. Configure the rule for TCP protocol, local port 80 (specific port) and Allow traffic (All ports as Remote port). See overview of expected result in this picture from Ian Parramore’s blog.
  6. Repeat steps on other ADFS/WAP machines.
  7. Configure HLB to do health check probe using the http protocol and this specific endpoint: http://servername-or-ip-address/adfs/probe

WORKAROUND #2 (RECOMMENDED): ADD A FALLBACK BINDING TO 0.0.0.0:443

It is possible to create a binding to make ADFS 3.0 and/or WAP to respond to SSL requests made out to ip-adresses. This is the recommended workaround as you can’t fully disable the SNI functionality completely. The process of adding this fallback binding consist of finding out the certificate thumbprint of your desired SSL certificate (added with private key to the Local Computer\Personal store) and then binding that thumbprint to all ip-adresses and for the ADFS or WAP application.

Follow these steps on all your ADFS 3.0 servers to add the fallback binding (and make your non-SNI compliant HLB be able to see your ADFS servers):

  1. Make sure that you have installed all available updates for Windows Server 2012R2 after adding and configured the ADFS STS or WAP Proxy role.
  2. Open a Command Prompt as administrator
  3. Run the following command:
    netsh http show sslcert
  4. You will see a list of SSL Certificate bindings.
  5. Mark and copy the ‘Certificate Hash’ value. Paste it to a temporary place (i.e Notepad)
  6. Mark and copy the ‘Application ID’ value. Paste it to the temporary place. The Application ID is what will associate the binding with ADFS 3.0 (for the internal STS servers) and WAP (for the ADFS Proxy).
  7. Now run the following commmand, where you insert the noted ‘Certificate Hash’ and ‘Application ID’ values from above (keep the { } characters):
    netsh http add sslcert ipport=0.0.0.0:443 certhash=Insert_Certificate_Hash_Here appid={Insert_Application_ID_here}
  8. Restart machine and repeat steps for remaining ADFS 3.0 machines.

After adding the fallback binding, your HLB should be able to reach your ADFS servers normally, and you can continue to configure and fine tune your load balancing. Be aware that if you replace your certificate in the future (when it expires), you must perform these steps again (a new certificate means a new certificate hash) – worth mentioning in your system documentation.

SUMMARY AND DISCLAIMER

Adding a http health check probe or a fallback binding can help us fix our load balancing, but my hope is that SNI support in HLB devices will become more of a standard than it is today. Like stated in the article, looking for native support with a software/firmware update and proper documentation from your HLB vendor is always recommended before applying the fallback binding workaround. If you are aware of SNI-related knowledge base articles and software updates from HLB vendors, please share them with me in the comments field below or on Twitter (@JesperStahle) and I will add them to this article.

Hands on with Office 365 Message Encryption (OME) using One Time Passcode

ARTICLE UPDATED OCTOBER 2014 AFTER LAUNCH OF OTP CAPABILITIES IN OME.

INTRODUCTION

At the beginning of 2014, Microsoft released new capabilities for email encryption, called Office 365 Message Encryption (OME). The functionality was first announced in november 2013 in this article. In October 2014, OME was improved drastically as the receiver of encrypted emails now can use a One Time Passcode (OTP) to access the encrypted message (documented here). At launch of OME, all receivers had to authenticate with either an Organizational Account (Azure AD) or a Microsoft Account (formerly LiveID) to access the email, which often led to confusion for external receivers. Now, the receiver can instead request a One Time Passcode to the same email address as to which the encrypted message was sent, and use that (within 15 minutes) to access the encrypted email. This post was updated in October 2014 to include this new OTP-functionality.

In this post, I will demonstrate the user experience of Office 365 Message Encryption, both for the end-user (using OTP) and the administrator (setting up IRM Licensing and all necessary Transport Rules etc).

SCENARIO

We will create a transport rule that will enable Office 365 Message Encryption on messages with a Sensitivity level set to Confidential.

STEP 1 – ENABLE IRM LICENSING

If you attempt to use Office 365 Message Encryption before first enabling IRM licensing, the operation will fail and give you this message:

You can’t create a rule containing the ApplyOME or RemoveOME action because IRM licensing is disabled.

To remediate this, we need to enable IRM licensing using the Admin portal and PowerShell. Follow these steps:

  1. First, enable RMS in your tenant by logging on to your Office 365 Admin Portal, navigate to Service settings (left pane). select Rights management (top bar), click Manage and finally hit the button Activate. A message should appear stating that RMS is activated for the tenant.
  2. The remaining steps will be done in PowerShell. Open Windows Azure Active Directory Module for Windows PowerShell, found here:
    http://technet.microsoft.com/library/jj151815.aspx
  3. Connect your PowerShell session to your Office 365 tenant an Exchange Online by entering the following:
    $msolcred = Get-Credential (Answer the credential prompt with your Office 365 tenant administrator credentials) 
    Connect-MsolService -Credential $msolcred

    $Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $msolcred -Authentication Basic -AllowRedirection
    Import-PSSession $Session
  4. When connected to Exchange Online, you can enable IRM licensing with just a few steps. The first step is to set the RMS Online key sharing location. You will use different configurations depending on where your tenant is located (North America, European Union or the Asia-Pacific area).Enter the command that matches your tenant location (choose one):
    North America: Set-IRMConfiguration -RMSOnlineKeySharingLocation https://sp-rms.na.aadrm.com/TenantManagement/ServicePartner.svc
    European Union: Set-IRMConfiguration -RMSOnlineKeySharingLocation https://sp-rms.eu.aadrm.com/TenantManagement/ServicePartner.svc
    The Asia-Pacific Area: Set-IRMConfiguration -RMSOnlineKeySharingLocation https://sp-rms.ap.aadrm.com/TenantManagement/ServicePartner.svc
  5. After setting the key sharing location, the next step is to import the Trusted Publishing Domain (TPD). Do so by entering the following:
    Import-RMSTrustedPublishingDomain -RMSOnline -name “RMS Online”
  6. The final step is to activate the internal IRM licensing. Do so by entering the following:
    Set-IRMConfiguration -InternalLicensingEnabled $True
  7. After enabling IRM licensing, verify the functionality by entering:
    Test-IRMConfiguration -RMSOnline

The test should pass with the result OVERALL RESULT: PASS.

NOTE: Enabling IRM licensing might need several hours for the change to take effect. If possible, allow 8 hours to pass before proceeding with STEP 2.

STEP 2 – CREATE THE OME TRANSPORT RULE

We will create a transport rule that enables Office 365 Message Encryption if the message is sent to a recipient outside the organization and the Sensitivity header have been set to Confidential.

Follow these steps:

  1. Log in to your Office 365 Admin Portal and navigate to Exchange Control Panel (Admin\Exchange).
  2. Navigate to Mail Flow, click the + icon and select Create a new rule…
    Newrule
  3. Give the rule a suiting name and click More options…
    moreoptions
  4. From here you can set your condition as it fits your needs, but for this example we will inspect the Sensitivity header and apply Message Encryption based on that. To do so, select the following conditions:
    Apply this rule if… A message header includes any of these words
    amessageheader
  5. Complete the Apply this rule if-condition by clicking the properties Enter text and Enter word so that the condition makes ‘Sensitivity’ header includes ‘Confidential’
    sensincl
  6. Click Add condition and select The recipient… Is external/internal. Click Select one… and select Outside the organization and hit OK
    outside
  7. Proceed with clicking Do the following… and select Modify the message security… and select Apply Office 365 Message Encryption
    applyo365m
  8. Hit Save at the bottom of the New rule editor. (If you get the message that IRM licensing is not enabled and have completed STEP 1 – please allow more time for the change to take effect, as stated in the Note after STEP 1).

STEP 3 – SEND A CONFIDENTIAL MESSAGE WITH OME

We will create a message with the sensitivity level set to Confidential and send this to a recipient outside our organization. Our transport rule will apply Office 365 Message Encryption to the message.

Follow these steps:

  1. Open Outlook or Outlook Web App. Both clients have the native functionality to set the sensitivity header. In the example, I will use OWA, but the procedure is the same in Outlook.
  2. Compose a new message. Enter an external recipients, give the message a subject and some content, then click  and Show message options…
    messageopt
  3. Set the message Sensitivity level to Confidential and hit OK
    confidential
    NOTE: In the rich Outlook client, Sensitivity options are found under Message Options\More options\Sensitivity.
  4. Send the message.

STEP 4 – RECEIVE AND OPEN THE ENCRYPTED MESSAGE

  1. Open the inbox of the external mailbox and find the encrypted message
    protectedmessage
  2. As you can see, the message includes an attached HTML-document. Open the attachment.
  3. To be able to read the encrypted message, you can choose to sign in with a Microsoft Account or Corporate Credentials (AAD/Office 365 account) that has a username that matches the e-mail address to which the encrypted message was sent – or (new October 2014) you can choose to open the message with a One Time Passcode that is sent to the same email address that received the encrypted message (html file). I recommend using Organizational Accounts when sending to other Office 365 users, as they can then open the message with their corporate credentials or even an existing session cookie (Single Sign-on experience) – but for external users, which we are demonstrating in this case, One Time Passcode would be prefered. I am selecting the OTP choise by following the link at the bottom (“Don’t want to sign in? Get a one-time passcode…”).
    Select "Get a one-time passcode..."
  4. Now, go back to your inbox (recipient) to find the One Time Passcode and a reference code (handy if you have received multiple encrypted messages at once to differentiate the OTPs for different messages). Make a note of the OTP and go back to the browser.
    Find the OTP in your inbox
  5. Enter the Passcode from the email in the Passcode field, then click Continue.
    step2
  6. The message will now load and be displayed in the browser.
    voila

CONCLUSION

The Office 365 Message Encryption is a very good feature that allows Office 365/Exchange Online customers to send encrypted messages both inside and outside their organization. The new OTP functionality makes it even better and more complete. This post demonstrated the basic capabilities of OME, and you can make even more advanced Mail Rules, such as applying RMS templates to the message if it is sent to an internal recipient, and use Office 365 Message Encryption only to external recipients – just configure the rule to fit your needs.

You can also customize/brand the message that includes the HTML-file by using the
Set-OMEConfiguration cmdlets. The cmdlets allow you to set custom e-mail text, disclaimer, portal text and even include your company logo for branding.

 

Can’t login to Lync Online on Domain Joined Machine in a Single Labeled Domain (SLD)

BACKGROUND

  1. You are using Lync Online together with Lync or Lync Basic client.
  2. You’r  client is joined to an Active Directory domain with Single Label Name (SLD). FQDN Name is, as an example, “domain” and not “domain.local”
  3. You can logon to Lync using mobile clients or from other clients than are not domain joined to your Single Labeled Domain.
  4. When logging on with the Single Labeled Domain joined machine, the Lync Client gives the following error:
    The Server is temporarily unavailable. If the problem continues, please contact your support team.

CAUSE

Single Labeled Domains are not supported with Lync Server, but there is no documentation regarding Lync Online instead of an on-premise Lync installation in combination with a client running from a Single Labeled Domain.

The fact is that this combination does not work wihtout additional configuration (see Resolution). Allthough the Lync Online infrastructure is not running in a Single Labeled Domain, the client can’t login from a machine joined to a Single Labeled Domain.

RESOLUTION

To solve this, all users that are running Lync Online must have a proper UPN-suffix in Active Directory. There is NO NEED to login with the UPN-name, but the attribute itself must be populated.

Follow these steps to verify the solution with a few users, before applying to all users:

  1. In you Active Directory forest, add an additional UPN Suffix. As suggestions to what to add, you can add the same as your SMTP domain(s), or just use the current SLD and add “.com”. Instructions on how to add UPN Suffix can be found here: http://support.microsoft.com/kb/243629
  2. On your own (an/or other) user account(s), change the UPN Suffix to the newly added “domain.com-like” using Active Directory Users and Computers. You find the UPN suffix on the Account tab, the right drop-down box under “User logon name”
  3. Wait for AD Replication to complete (15 minutes give or take, depending on number of AD sites etc.)
  4. Log off and on to your workstation
  5. Attempt to log in to Lync again
  6. Repeat for remaining users

With a correct UPN Suffix, Lync Online will allow you to log on from the domain joined machine, regardless if you logged in with your samAccountName or the full UPN-name.

Shared Accounts (POS/Kiosk) and Single Sign-On in the Same Environment

BACKGROUND

You have successfully configured Single Sign-on in Office 365 with the following conditions:

  1. ADFS is deployed with one internal pair (STS) and one external pair (Proxy), both pairs are load balanced. ADFS is running at least version 2.0 with the latest Update Rollup installed.
  2. Clients that resides on the internal network have the ADFS URL (in this example sts.Company.com) pointing to the ADFS STS server pair (the LB VIP).
  3. Clients that resides on the external network have the ADFS URL (sts.Company.com) pointing to the ADFS Proxy pair (the LB VIP).
  4. On all Active Directory-domain joined clients, the ADFS URL (sts.Company.com) is added to the Local Intranet Zone using a Site-to-Zone Assignment List GPO.
  5. You have not configured ADFS to allow Single Sign-on for non-IE browsers (Disable Extended Protection).
  6. You have verified that Single Sign-on works as expected by navigating to http://outlook.com/OWA/company.com (where Company.com is a custom domain that is configured as a Federated Domain in your Office 365 tenant). When navigating to the URL, you are redirected to Outlook Web App without being prompted for credentials.

This scenario is exactly what we want for our users that are logging on to Windows everyday using their personal Active Directory account. It is a seamless solution that allow the user to consume Office 365 services without entering credentials multiple times through-out the working day.

The problem is if parts of the organization is using common/shared Active Directory accounts for logging on to Windows. A few examples of when common Active Directory accounts are used for logging on to Windows are at: Point of Sale terminals (POS), front desk/receptions, and at heavy industry operation sites (factory users are restricted by gloves/outfit).

These users will not be able to access their personal Office 365 services as they are logged in to Windows with the shared account, and when attempting to access, for example, Outlook Web App, they either get a message that the current account does not have a valid license (if the shared account is not licensed), or they will be logged in to the wrong mailbox (if a licensed user is logged on to Windows when the other user is taking over the session). The user have no chance to login as themselves to reach their personal mailbox etc, as we have configured Single Sign-on to Office 365, without any credential prompts allowed.

In summary, we have a situation where users logging on with personal Active Director accounts to their personal computers have a perfect experience with Single Sign-on, but our users without personal computes that are logging on to Windows with shared accounts, are simply not able to reach their mailbox and other data without first logging in to Windows with their personal account – which is not feasible for the business operation.

CAUSE

This bevaviour is fully expected. It is not ideal for users to share Active Directory accounts for loggin on to Windows, both from the security perspective (complicance and auditing) and from the functional perspective (single sign-on and personal data access). To remediate it, the general recommendation is of course to have all users logging on to Windows with personal Active Directory accounts, but in reality, this is not always possible. The reasons are many, and they are not described on TechNet or in any text book examples, but they are usually related to the time it actually takes to log off and log back on. At a POS, every lost second can be a lost sale, and at the factory floor, every lost second can be a compromise of a persons safety.

Technically, our Single Sign-on configuration is doing exactly what it is supposed to do. A user attempts to reach Outlook Web App, get’s redirected to ADFS STS and authenticates using the Kerberos TGT-ticket issued for the shared account when logging on to Windows. ADFS grants a Token, including claims for the shared account. The Token is then presented to Office 365, who translates the claims to either a non-licensed user (giving the No license error), or if it exist, to the mailbox for the shared account (loading the wrong mailbox for the user).

RESOLUTION

To remediate this problem, without affecting the Single Sign-on experience for our Office workers with personal computers, we must break the Single Sign-on configuration only for the shared accounts.

To accomplish this, we will configure the Site to Zone Assignment List GPO.

As stated in the Background above, the ADFS URL sts.Company.com is added to the Local Intranet Zone to allow Single Sign-on. If we remove it from there, the user will not be able to present the Kerberos TGT-ticket when logging on, and we will prompted for credentials again. This is a bad user experience and should never be the routine for Office users with personal computer and personal accounts, but such credential prompt is what can allow a user that has logged on with a shared user account to actually login to Outlook Web App (or other Office 365 Service) with a personal account and reach the personal mailbox.

High-level description of the solution:

  1. Depending on how Group Policy is used in your organization today, first decide if this setting should be computer based or user based. Both can be applied for this.
  2. Configure separate Site to Zone Assignment List GPO’s for personal Active Directory accounts and for common/shared Active Directory accounts. (Or for personal computers or shared computers, depending of outcome in #1).
  3. For the personal account/computer GPO, place sts.Company.com in the Local Intranet Zone – this will give them Single Sign-on
  4. For the shared accounts/computers, Place sts.Company.com in the Trusted Sites zone – this will break Single Sign-on and give the user a credential prompt on access to the services. The user can respond to the credential prompt with personal credentials and by that accessing the personal mailbox/data.

Here is a good overview of the Site to Zone Assignment List GPO: http://blogs.msdn.com/b/askie/archive/2012/06/05/how-to-configure-internet-explorer-security-zone-sites-using-group-polices.aspx (Scroll down a bit).

COMMENT: Some organizations solves this issue by letting all clients redirect sts.company.com to the ADFS Proxy even internally. This methdo will also present the credential prompt, but it will affect all users – including those that are using personal accounts on personal computers – which is a bad user experience for them. Using GPO and Site to Zone assignment is  a better way to go.

PST Capture Tool 2.0 Only Imports Deleted Items Folder

BACKGROUND

You are attempting to import data to Exchange Online mailboxes using the PST Capture Tool 2.0 (http://www.microsoft.com/en-us/download/details.aspx?id=36789). The following works:

1. You can import PST files to the tool
2. You can map the PST files to mailboxes in Exchange Online (x64 bit version of Outlook is required to see the mailboxes)
3. If you open the PST file in Outlook on the machine, you can see all the content

When starting the Import, the stage called “Transferring” completes 100%. Then, when the Import stage begins, you can see that it only finds 2 folders. The progress shows Importing Folder 0/2, 1/2 and then 2/2.

After the import is done, you can see in the Exchange Online mailbox that the only content that have been imported is the Deleted Items folder.

CAUSE

This occurs if the PST file that is imported is in an incompatible format. Without going into, or knowing, the details of it all, PST files can be created in different structures. The version of Outlook together with how PST Capture Tool 2.0 works decide if all content in the PST files root folder (IPM_SUBTREE) will be visible to the operation or not. Read more here: http://msdn.microsoft.com/en-us/library/ff387699(v=office.12).aspx

The error can occur when the PST file have been exported from Exchange Server 2010 SP3 using New-MailboxExportRequest.

RESOLUTION

Run SCANPST.EXE from the Program Files\Microsoft Office\(VersionNumber) folder on the machine. Use SCANPST.EXE to scan and repair the PST files you want to import. After repairing the file, PST Capture 2.0 will import all folders and objects.

This process takes a while, and if you are to do 100+ mailboxes, it might have a severe impact to your planning. Depending on the PST file, it might help to change Outlook version on the machine where you run PST Capture 2.0 – just make sure that it is always x64 bit.

Free/Busy not working from Office 365/Exchange Online to On-Premise Exchange / Hybrid Configuration – “The external recipient’s server could not be determined” “No information” (You forgot to iisreset)

BACKGROUND

You have successfully deployed a Hybrid Configuration between your on-premises Exchange 2010 SP3 Organization and Exchange Online. Everything works as it should, except for Free/busy look-ups made from Exchange Online mailboxes to a mailbox that still resides on-premise.

You have verified that the following is OK:

  1. You have run ExRCA diagnostics using https://www.testexchangeconnectivity.com/ , selected the Office 365 tab and then Free/busy tests. You simulate Free/busy lookup from Online to On-premise and it works with all green lights.
  2. You have also followed all the troubleshooting steps presented in this Microsoft Knowledge Base article: http://support.microsoft.com/kb/2555008 – all with successfull results.

Still, when attempting to do Free/busy lookups from Online to On-premise, you get the message in Outlook or OWA saying “No information” with the reason “The external recipient’s server could not be determined”.

CAUSE

Settings made during your Hybrid Configuration and/or extended troubleshooting of this error have not been fully commited to Internet Information Server (IIS). Allthough the diagnostics (ExRCA etc) say that your configuration is right, vital configuration is not yet effective.

RESOLUTION

First step in resolving Free/busy errors is to follow the two steps mentioned in the Introduction to this post (ExRCA diagnostics and following KB2555008 – see above). If that does not work:

Doing iisreset on your CAS server publishing EWS/Hybrid could fix this issue.

Plan for a very short outage in Exchange services (OWA/EWS etc) and open the command prompt on your Hybrid CAS server. Then run the command iisreset

After doing IISReset, Free/busy lookups from Online to On-prem should start working within minutes.

I have seen this in the field where we spent days troubleshooting this issue on Exchange 2010SP3 hybrid configurations after finally solving it with this basic, but classic, method – restarting the services. There could of course be other reasons behind the error (i.e related to how Autodiscover is published to the internet), iisreset should not be considered as a “general cure”, but a necessary troubleshooting step if your configuration is done by the book and it still does not work.

“Enable Centralized Mail Transport” / “Route all Internet-bound messages through your on-premises Exchange servers” and NDR Remote Server returned ‘550 5.7.1 Unable to relay’

BACKGROUND

You configure the Hybrid with the option to route back e-mail from Exchange Online users to your On-premise organization Before sending them to the Internet.

  • In the Exchange 2013 Hybrid Configuration Wizard, this option is called: “Enable Centralized Mail Transport”
  • In the Exchange 2010 Hybrid Configuration Wizard, this option is called: “Route all Internet-bound messages through your on-premises Exchange servers”

This is not the default option, but is necessary for some organizations due to compliance or message tracking reasons.

What it does is that it is adding a wildcard (*) in the send connector in FOPE/EOP, that normally only is used for your shared domains, to route back e-mail to users On-premise that have not been migrated to Exchange Online. The wildcard enables this send connector for all domains on the Internet, thus route the e-mail back to your On-premise Exchange Environment.

Your On-premise Exchange Environment can without any problem route e-mail to your internal, not yet migrated users, because that Exchange Environment knows about those domains and their recipients. Internet recipients, on the other hand, will require your message from the Exchange Online user to be RELAYED outside your On-premise Exchange, back to the Internet.

This fails with the following NDR (bounce message) information to the sender in Exchange Online:

Remote Server returned ‘550 5.7.1 Unable to relay’

CAUSE

The receive connector that the Hybrid Configuration Wizard creates on-premise to Catch the e-mail that Exchange Online sends back is not fully configured and ready for use after running the Hybrid Wizard.

RESOLUTION

To fix the “Inbound from Office 365” receive connector, you need to both populate the list of allowed IP-addresses and enable the receive connector for relaying.

Follow these steps:

  1. Open the Receive Connector properties in EMC or ECP (2010 or 2013 on-premise)
  2. First, open the Network tab/settings to investigate the list of remote servers, “Receive mail from remote servers that have these IP addresses”.
  3. The list of allowed IP-addresses is supposed to be populated by the Hybrid Configuration Wizard – BUT IT IS NOT COMPLETE! Use this article to verify each address space is included in this list: http://help.outlook.com/en-us/140/gg263350.aspx . Make sure to read the complete article to understand that the list of IP-addresses is subject to change etc.
  4. After making sure that your list of allowed IP-addresses is complete, you still need to enable the receive connector to allow further relaying to Internet recipients when the message origins from Exchange Online:
  5. Fire up Exchange Management Shell and run this PowerShell command to enable the “Inbound from Office 365” receive connector for relaying:

Get-ReceiveConnector ” Inbound from Office 365 ” | Add-ADPermission -User “NT AUTHORITY\ANONYMOUS LOGON” -ExtendedRights “Ms-Exch-SMTP-Accept-Any-Recipient”

After filling up the list of allowed IP-addresses and enabled it for relaying to Internet recipients, your mail flow will work as expected with the “Centralized Mail Transport” / “Route all Internet-bound messages through On-premise” option enabled.

Note: Be careful not to add excessive IP-address ranges to your receive connector. You do not want to create an open relay – make sure that you only allow Exchange Online hosts to relay.

 

 

This mailbox exceeded the maximum number of large items that were specified for this request. (Fatal error TooManyLargeItemsPermanentException has occurred.)

POST UPDATED TO REFLECT THE EXCHANGE ONLINE ONBOARDING SIZE LIMIT INCREASE (FROM 35 to 150 MB) ANNOUNCED JANUARY 2015 (INFO).

BACKGROUND

When moving mailboxes to Exchange Online, the Remote Move Request fails with the following error message in Exchange Management Console (EMC):

Error: This mailbox exceeded the maximum number of large items that were specified for this request.

When viewing the log file for the mailbox migration, you see the following details:

Fatal error TooManyLargeItemsPermanentException has occurred.

CAUSE

The message size limits for Exchange Online migrations is 150 MB. It used to be 35 MB prior to January 2015, but have now been increased (info). If larger items than 150 MB are attempted to be moved through a default Remote Move Request initiated from the EMC wizard, the job will fail with the error message above because Exchange Online refuses the messages due to their size. If you have mailboxes containing messages that are +150 MB in size, you can still migrate these mailboxes but you will need to tell Exchange that it should simply skip the large messages that are refused and not abort the complete move request when encountering them.

RESOLUTION

If a mailbox has items that are 150 MB+ in size, you can solve this error in two ways:

  1. Either instruct the end-user to remove the item. Send an instruction on how to sort items on size in Outlook and tell them to Shift+Del Everything above 150 MB in size. Elite users can maybe even handle an instruction on Search Folders/Filter. After the users have removed the items, remove the Move request and start over again (resuming it will fail).
  2. Or, allow the move request to skip the 150 MB+ items that are refused by Exchange Online and continue the migration without aborting the entire job. Here is how:

Unfortunately, you cannot use the EMC wizard for this, you will need to use the Exchange Online Control Panel (where the migration wizard will allow you to adjust the parameters) to run the migration or use PowerShell. Using the Exchange Online Control Panel works for both Exchange 2010 and Exchange 2013 hybrid configurations.

2.1: USING THE EXCHANGE ONLINE CONTROL PANEL INSTEAD OF EMC (RECOMMENDED)

The most simple way to solve this problem is to run the migration from the Exchange Control Panel (as said, this works with both Exchange 2010 and 2013 hybrid configurations). Follow these steps:

  1. Log in to you Office 365 Admin Portal and navigate to Exchange Control Panel (Admin\Exchange).
  2. Navigate to Recipients (left pane) and then the tab Migration (top pane/tabs)
  3. Click + (plus) and Migrate to Exchange Online
  4. Select the first choice to initiate a remote mailbox move
  5. Select users to migrate or use a CSV to create a new migration batch
  6. At the next step, enter your MRS endpoint, that is where EWS is published (your hybrid server). Example: ews.company.com
  7. At the next screen, you can give the batch a name for tracking and logging purposes, select if you want to migrate an Archive Mailbox, and most importantly set the allowed number of large and corrupt/bad items to be skipped for the migration. Increase this number to be able to run the migration that failed. As an example, you can set both values to 50.
  8. At the next screen, enter who will have reports when the migration batch is completed, set scheduling options (start now or later) and select if the migration should be completed directly or if it should be automatically suspended when the mailbox replication have reached 95%, so that you can switch over the user when you want in a controlled manner. If using the auto suspend option, the user will not notice anything until you choose to complete the migration, then the last 5%/delta will be synchronized and the user will be switched over (email will be forwarded to Exchange Online mailbox and Outlook will re-configure for use with the Exchange Online mailbox etc).
  9. When completing the wizard, you can then track the migration from the Migration view in the control panel. If you are using automatic suspend, here is also where you can complete the migration after reaching 95% (mark the job and select complete).

2.2: USING POWERSHELL

As stated above, using Exchange Online Control Panel is recommended. If you would like to use PowerShell instead, follow this method.

If you are moving just one user, follow these steps:

New-MoveRequest -remote -Id firstname.lastname@smtpdomain.com -RemoteHostName “ews.companydomain.com” -RemoteCredential $Cred -TargetDeliveryDomain ‘tenantdomain.mail.onmicrosoft.com’ -SuspendWhenReadyToComplete -LargeItemLimit 50 -BadItemLimit 50

Note that I added LargeItemLimit and BadItemLimit. These parameters will allow up to 50 large (over 150 MB) messages to be refused and skipped, and up to 50 corrupt messages to be skipped as well.

If you are moving several users, you could use a CSV file with a column that says UPN, filled with your migration batch users UPN-names:

Import-Csv C:\MigrationBatch.csv | foreach {New-MoveRequest -Identity $_.UPN -Remote -RemoteHostName ews.companydomain.com -TargetDeliveryDomain ‘tenantdomain.mail.onmicrosoft.com’-BadItemLimit 50 -LargeItemLimit 50}

For both single and bulk, consider adding the following parameter to use the AutoSuspend feature:

 -SuspendWhenReadyToComplete -Suspend

Moving mailboxes with EMC is very convenient, but it will not give you the option to allow corrupted or large (150 MB+) items to be skipped and overseen in the migration job. Go Exchange Online Control Panel or PowerShell to save the day.

Certificate referenced by property OrgPrivCertificate in the FederationTrust object is expired. (Be PATIENT!)

BACKGROUND

You are running this command to see if you are on top of your Hybrid Configuration or not:

Test-FederationTrust | fl

It looks pretty good, but fails at the end with this depressing statement:

Id      : OrganizationCertificate
Type    : Error
Message : Certificate referenced by property OrgPrivCertificate in the FederationTrust object is expired.

 Naturally, you investigate the Exchange Delegation Federation Certificate on your side and find that is good for another five years! So why is it telling you that it is expired?

CAUSE

If you start investigating this issue, you will find pointers to ADSIEdit and how to cycle up new certificates etc. My advice: Don’t go there. If this configuration is against Office 365/Exchange Online/Microsoft Federation Gateway – it is probably more simple than that. Microsoft Federation Gateway (MFG) is very sensitive when it comes to time.

RESOLUTION

Notes from the field:

  1. MFG is sensitive about time. Investigate your time zone settings, external time source (AD PDC Emulator settings and VM hypervisors etc.)
  2. If time settings are OK, just WAIT!

Waiting will solve this problem. For the configuration that made me write this blog post, it took 45 minutes, but I suggest that you should wait as long as time allows you. Just leave all remote PS sessions for the Office 365 tenant in question for a couple of hours and try again. The result after waiting  (no configuration made in the meantime):

[PS] C:\>Test-FederationTrust | fl

Id      : FederationTrustConfiguration
Type    : Success
Message : FederationTrust object in ActiveDirectory is valid.
Id      : FederationMetadata
Type    : Success
Message : The federation trust contains the same certificates published by the security token service in its federation metadata.
Id      : StsCertificate
Type    : Success
Message : Valid certificate referenced by property TokenIssuerCertificate in the FederationTrust object.
Id      : StsPreviousCertificate
Type    : Success
Message : Valid certificate referenced by property TokenIssuerPrevCertificate in the FederationTrust object.
Id      : OrganizationCertificate
Type    : Success
Message : Valid certificate referenced by property OrgPrivCertificate in the FederationTrust object. Id      : TokenRequest
Type    : Success Message : Request for delegation token succeeded.
Id      : TokenValidation
Type    : Success
Message : Requested delegation token is valid.

 

After cracking one of these, you really feel like an Exchange genius. Awesome.