Managing Change in Enterprise Office 365 Implementations (or “Do We Need a Test Environment?”) – the Lost Documentation

BACKGROUND AND PURPOSE

One of the top challenges for Enterprise customers in their journey to Office 365 and Cloud based services is the transformation needed to handle the constant service Changes that make out a natural part of SaaS and the Cloud. The concept of “evergreen services” is still for many completely new, and customers often finds this parameter more challenging than the technical implementation and onboarding itself. Microsoft recognizes this, and are committed to help organizations prepare for change by implementing various change control mechanisms to the service. They are also more transparent to their customers than they have ever been in regards to Product roadmaps (see the Office 365 Public Roadmap here) and Community presence (see the Office 365 Network on Yammer here). Regardless of all these efforts from Microsoft, Enterprise customers that are new to Office 365 always asks me and my team for advice in this complex topic. The question is initially almost always formulated in the same way: “How do we create the test Environment for this?”. The answer: “It’s not that simple…”, and then the discussion follows, uncovering all aspects from handling operational maintenance to planning for unavoidable updates in the user interface.

The purpose of  this post is to give an overview to how Enterprise customers should comprehend and plan for ongoing change after Office 365 is implemented and all the users are onboarded to the service. Microsoft provides good information in their service descriptions, on the Office Blogs etc, but this post is to be considered as “the lost documentation” of the holistic overview that often is requested by customers when they are new to Office 365.

OFFICE 365 IS “EVERGREEN” – WHAT COULD POSSIBLY GO WRONG?

The concept of Evergreen means that the service will constantly improve and evolve, without you as a customer having to worry about it. The reality of it is that you do not need to take care of server version upgrades, security patches, feature implementation or maintenance – your service provider takes care of that for you. This will save you time and money, not least as you never have to run an upgrade project again for i.e Exchange Server or Skype for Business Server etc. On the other hand, if your organization is not used to rapid releases and always being on the latest version, you will need to shift your mindset from “big change projects every five years” to “smaller updates every month”. The concept is often referred to as moving from “waves to ripples”, a good metaphor to how things will change after the move to Office 365.

Evergreen is great, however it will put new requirements on your IT-organization if you are not used to deliver changes rapidly. This scares many Enterprise customers, and sadly some are disqualifying Office 365 just because they think that they can’t control change in Office 365, and that “a high release pace determined by somebody else” is therefore nothing for them. That is a false apprehension, as there are many ways to control change in Office 365, which will be covered in this post.

As the World is changing and millennials are entering the workplace, the pressure from the business on the IT-organization to deliver a modern digital workplace is often very high. Also, providing modern productivity- and collaboration tools is often key to prevent rogue IT, where users are adopting Commercial solutions for cloud storage, or send sensitive information to Commercial e-mail services so that they can access it anywhere on any device etc. These business requirements goes hand in hand with what Office 365 offers, so the IT-organization will need to deliver, and at the same time manage change in an Evergreen service. Let’s have a look how this can be accomplished in practice.

BECOME AN “EVERGREEN READY ENTERPRISE” WITH OFFICE 365 – CATEGORIES OF CHANGE

I will explain how to manage change in Office 365 by breaking the topic down to categories. I will explain the challenges and remediation options for each category. If you have ideas for other categories, please reach to me on Twitter (@JesperStahle), and I will consider adding them to this post. My intention is to keep the documentation alive and updated as Office 365 is changing.

CATEGORY #1: THE ROLE AND RESPONSIBILITY OF THE SERVICE OWNER

CHALLENGE: The shift to become “Evergreen ready” starts with the role of the Service Owner (sometimes referred to as Service Responsible or Service Manager, depending on how ITIL-influensed operations model you might have). The person owning the Office 365 service, regardless if it is the whole suite, or independent services (such as Exchange Online, SharePoint Online, etc.), must be prepared to become proactive in a whole new way than before. Traditionally, the Service Owners might find themselves in a situation where there are several months or even years between service changes, and the role is often described as more reactive than proactive. When a change is needed due to new dependencies or business requirements, the Service Owner would be notified and take the necessary decisions and issue transformation projects.

When the Service Owner becomes responsible to deliver an Evergreen service within agreed SLAs, it will be required that he or she becomes more pro-active to be notified of upcoming changes, before they are implemented and affects end users and the IT-operation. If the notifications are missed or ignored, the Service Owner will one day have a change issued outside of their control, which might cause frustration from both IT-operations and end users, and in the worst case even impact the SLA for which the Service Owner is responsible for.

BECOMING PROACTIVE ENOUGH: It is vital that the Service Owner of Office 365 suite, and/or its components, becomes proactive by monitoring all the communications from Microsoft regarding upcoming changes. Microsoft will not implement changes in Office 365 without first notifying their customers, but if you do not have a proper ownership and process for reading and responding to these communications they are to no use for your organization. It is a bad practice to trust the administrators of Office 365 to monitor the communications about upcoming changes and react to them accordingly – the administrators should put their focus on administering the service and enabling new or changed features as that is ordered/issued by the Service Manager. A better practice is to have a named Service Owner that takes responsibility for service communications from Microsoft as well as monitoring the Office 365 Roadmap to plan for upcoming changes and feature introductions.

In practice, this means that the Service Owner should monitor four main communication channels and implement processes to be able to rapidly respond to the changes announced in them. Here is the list of the four main communication channels:

  1. The Office 365 Message Center: In the Office 365 administration portal there is a Message Center that is used to notify customers about upcoming change and incidents. The Service Manager can access the Message Center by acquiring an administrative role in the Office 365 tenant (i.e the Service Admin role) then visit the Office 365 Admin Portal and navigate to Message Center. Here is a direct link. Here is an example screenshot of how the Message Center looks like (click picture to enlarge):The Office 365 Message CenterAs you can see, there are different categories of information. Not every communication in the message center is necessarily targeted for the Service Owner or similar function. Here is an overview of the Message Center taxonomy (click picture to enlarge):The Message Center TaxonomyIt is up to every organization to shape their own RACI matrix for these communications, but it is generally recommended for the Service Owner to be responsible for communications classified as “Stay informed” and “Plan for change”, and to be informed about communications classified as “Prevent/Fix issue” and “Service Incidents”.
  2. The Office 365 Public Roadmap: The next communications channel the Service Owner should be responsible to monitor is the Office 365 Roadmap, found here. This is a great evidence that Microsoft is aware that change requires planning. The roadmap site let customers see which features are in development and which that are currently being rolled out. When features are due to be rolled out, you will also be notified in the Message Center, so the main reason why the Service Owner should have the ownership and responsibility to act on this communication channel is for the organization to be aware about features that are in the “In development” category. By monitoring this category, the Service Owner can avoid surprises in the Message Center, and will have much more time to plan for change. Here is an example screenshot of how the Office 365 Public Roadmap looks like, note the category “In development” (click picture to enlarge).Office 365 Public Roadmap http://roadmap.office.com
  3. The Office Blogs: The third communication channel every Office 365 Service Owner should monitor is the Office Blogs – found here. Here you will find various communications from the teams at Microsoft that build Office 365, not just feature announcements, but also interesting costumer stories, news round-up articles and video casts (Office Mechanics) covering different a wide variety of topics. A good advice is for Office 365 Service Owners to have the Office Blogs as their start page in their web browser.
  4. The Office 365 Network on Yammer: With over 60 000 members and growing, the Office 365 Network is a community for Office 365 customers and IT-professionals world wide. Service Owners can participate in various groups within the network to discuss Office 365 topics. Microsoft themselves has a strong presence here, and often customers find their questions answered directly by representatives from the Microsoft product groups. This is a great forum for Service Owners as they can learn how others are solving issues that they are facing. The recommendation for the Service Owner is to monitor the network (and/or it’s quarterly digests known as the Office 365 Network Bulletins), but it is not necessary to read all its content daily. The Office 365 Network is found here.

CATEGORY #2: PROCESSES FOR SERVICE UPDATES, FEATURE INTRODUCTIONS AND UI CHANGES

Release Options: In Office 365, updates and new features are released constantly. Staying informed through the communication channels helps you prepare for change, but in order to be fully prepared processes are required for testing and evaluating new functionality. Therefore, it is possible to nominate users in the organization to be part of the First release ring (wave) for updates. First release users will have new features and updates before other users, so that they can test the updates in production and prepare integrations and documentation, such as end-user communications.

It is also possible to nominate all users in the tenant to be part of the first release ring by enabling First release for scope of the Entire organization. That would give everyone access to new features sooner, but will give the Service Owner and the IT-organization less time to prepare for change, which is not optimal for Enterprise customers. In May 2015, around 10% of all Office 365 customers had enabled First Release for the Entire organization, not just for selected people.

Find more information about the First Release options here. Here is an example screenshot of the Office 365 release options and how they are configured in the Office 365 admin portal (click picture to enlarge).

Release Options in Office 365

End-user awareness: Upcoming change is announced through the communication channels, and will be available for test and evaluation for the users nominated to be part of the First release ring. Those individuals should in concert with the Service Owner evaluate all dependencies to each new feature and update, as well as assess if end-user communications will be necessary. Where end-user communication is necessary, Microsoft often provide templates and input for your organization, so you only need to review the material and make your finishing touches to it.

The Skype for Business example: A good example of a change process was when Microsoft changed the name and user interface to Skype for Business from what was formerly known as Lync. Microsoft first announced this in the Roadmap site and on the Office Blogs, then shortly after in the Message Center under the category “Prepare for Change”. All the information needed was provided for the Service Owner to plan for change, and a preview version of Skype for Business was made available to install for the First Release users (or anyone else who wanted to try it). Microsoft prepared end-user information in both written form and in video format (see picture below) for customers, ready to be distributed directly to end users. The article with that material can be found here for reference. If any customer had any questions about the update – before, during or after – they could ask them in the Office 365 Network on Yammer and have the discussion going, and a potential answer, moments later.

Example of end-user material provided by Microsoft when Lync became Skype for Business

CATEGORY #3: PROCESSES FOR OFFICE 365 PROPLUS UPDATES ON CLIENTS

Whatever you did before will work: A common misunderstanding is that if you deploy Office 365 ProPlus (Click-to-Run), you have no control of updates and your users will have features introduced without your control. The reality of it is that any change/update process you had for MSI-based distribution of Office can be used with Office 365 ProPlus Click-to-Run, you just have to adjust your distribution method a bit. You can distribute Office 365 ProPlus using SCCM (good information in this guide) and you can then distribute updates in a staged fashion with test rings/waves, just as you probably did before.

When deploying Office 365 ProPlus, you can use a configuration file (config.xml), or Group Policy settings, to configure the location for updates for individual computers using the UpdatePath parameter. Read more about the config.xml parameters here, and the GPO settings here. The UpdatePath will point to a file share on the network where binaries for the latest available build of Office 365 ProPlus will be located. It is up to every organization to plan and design the file share – you can use shared folders on SCCM Distribution Points, a DFS file share or individual file shares distributed on site-local file servers etc. You can have separate file shares for pilot groups that can evaluate the Office updates (you place new updates here first), and these updates can then be moved to the production file share that are used by the majority of the users. This creates an environment where you can stage the updates and evaluate them before distributing to the masses, which also facilitates end-user communications if needed for user interface changes etc.

Updates can be postponed, but to stay fully supported you should deploy Office 365 ProPlus updates within 12 months from when they were released. Security updates and feature updates are today bundled together, but options for separating them will be available soon. Regardless of that, the recommendation is to evaluate updated versions in the pilot ring immediately when they are available, and distribute them to the production ring as soon as possible after the verification. Consuming cloud services requires the client to be up to date to have full support, optimal performance and all features available – try not to fall into old habits and postpone updates just because you have the option to do so for 12 months.

First Release users (see Category #2) will have the option to download Preview versions of Office 365 ProPlus to test functionality that is due to be released in a bit more distant future. The First Release users can sign in to the Office 365 Portal and download the Preview version from the Software section (direct link here) – just scroll down on the page, the Preview version is located under the download section for the current version. The Preview versions will never be displayed for users that are not in the First Release ring.

An example overview of this can be found below. In the graphics you can see a file server hosting two file shares containing different versions of Office 365 ProPlus – one for the Pilot Ring and one for the Production Ring. The binaries (bits) are downloaded from Office 365 on the server (can be automated with scheduled task). Users in the Pilot Ring have their UpdatePath parameter set to the Pilot-share, and the users in the Production Ring have their UpdatePath parameter set to the Prod-share. In the First Release Ring, users are downloading Preview bits from the Office 365 Portal under the Software section. Click graphics to enlarge.

 

 

Overview of how Office 365 ProPlus updates can be distributed in the Enterprise.

CATEGORY #4: MANAGING CHANGES TO YOUR ON-PREMISES FOOTPRINT (ADFS, AADSYNC, EXCHANGE HYBRID, ETC.)

Enterprise Office 365 deployments are integrated with the on-premises infrastructure through an “identity bridge” (synchronization and federation, also known as AADSync and ADFS) and sometimes an Exchange Hybrid configuration. There can also be integrations with Lync/Skype for Business (Hybrid Configuration) as well as Search integrations between SharePoint Server on-premises and SharePoint Online.

Test environments for this can be provisioned, but it does not makes sense to do so if the on-premises system in question does not have a full test environment to begin with. As an example, if you don’t have a test environment for Active Directory before introducing the identity bridge to Office 365/Azure AD, it does not make sense to provision a test environment for the AADSync and ADFS components, as they would read and write to the exact same source and target as the production instance. If, however, you do have a full featured test environment for Active Directory, you could of course choose to provision a separate Azure AD tenant to which you integrate the test environment through an identity bridge in the test environment.

The same logic goes for Exchange, Lync/Skype for Business and SharePoint – if you don’t have proper test environments to begin with, it would not be possible to produce a fair test environment for the Office 365 integration.

Different support statements are true for the on-premises components. Here are the main guidelines to follow:

  1. AADSync / DirSync /AADConnect: The installation of the sync tool is meant to be “appliance like”. You do not necessarily need to update to every version that is released to stay supported unless you’re noticed to do so (all though it is generally recommended to stay up to date from the feature- and performance perspective). You can choose not to upgrade the sync tool until you are implementing a certain feature that requires a certain version – or until you are receive a message in the Message Center (see Category #1) saying that you need to upgrade to stay supported. The general recommendation is for the Active Directory Sync Tool Service Owner to monitor the Version Release History article, found here, to be aware about when updates are released and plan for installing the update accordingly. If test environments are not used, there are is a “safety net” measure that customers can take to prevent configuration errors in the sync tool to soft delete users in Azure AD. It is called PreventAccidentalDeletes, more information can be found here.
  2. ADFS: Like with Active Directory and Windows Server in general, it is recommended to keep ADFS updated through Windows Update. This is recommended for security reasons and to ensure optimal performance and functionality. So far, all versions of ADFS that have been released since Windows Server 2008 (ADFS 2.0) have been supported in Office 365, so you may consider your installation fully supported as long as you are installing all Windows Server updates from Windows Update, and as long as the version of Windows Server that you are running is in Mainstream Support (find out more here). However, new ADFS-features are available with new versions of Windows Server, which may encourage you to upgrade your ADFS environment as well. An example of such feature that have made many ADFS 2.0 (Windows Server 2008/2008R2) customers upgrade to ADFS 3.0 (Windows Server 2012R2) is the Extranet Lockout feature (read more here).
  3. Exchange Hybrid Configuration: To stay supported in an Exchange Hybrid Configuration, your on-premises Exchange Server environment must be on a supported version (at the time writing Exchange 2007 SP3RU10, 2010 SP3 and 2013 CU7, where 2010 and 2013 can be the Hybrid servers facing Exchange Online) and the latest available build minus one cumulative update should be installed – also referred to as the “N to N-1 guideline”). This means that if you have an Exchange Server 2010 SP3 environment on-premises, and the latest available version is Exchange Server 2010 SP3 Update Rollup 10, you would have to have at least Exchange Server 2010 SP3 Update Rollup 9 to be fully supported. The written support statement can be found here, and it actually does not mention the N to N-1 guideline. However, if you run in to issues and attempt to open a support ticket, Microsoft will remind you of the N to N-1 guideline and encourage you to update to the N to N-1 build and reproduce the issue on that version before troubleshooting continues.
  4. Lync, Skype for Business and SharePoint Hybrid Configurations: For these workloads, the support statements are not as clear as with Exchange etc. The general recommendation is to keep the on-premises server products on the latest available build (install available updates) and monitor the Message Center for notifications about supportability.

CATEGORY #5: DEVELOPMENT, CUSTOMIZATIONS AND BRANDING

There is one category where test tenants makes absolute sense – and that is for system development. In the perspective of Office 365 as a platform for developers, providing a test tenant is absolutely necessary. Everything from branding the suite bar to releasing code in SharePoint Online benefits from being tested first. However, these test tenants do not require a full identity bridge etc to do their job – instead developers can provision their own test tenants from their MSDN Subscription benefits (or even Trial plans), or you can provision one on a separate subscription. This test tenant can be completely out of scope for the Service Owners responsibilities.

SUMMARY AND DISCLAIMER

There are many misunderstanding when it comes to Office 365 and managing change in the Enterprise. Hopefully this article can help som organizations realizing that the Cloud, SaaS-apps and “Evergreen services” are manageable even in an Enterprise environment, and that good instruments exist so that the Service Owner can master even a very rapid release pace. Please note that the information given in this article is to be considered as guidelines shaped from real-world experience – but that change management is a complex topic that always have organization specific requirements. Modernizing change management to be “Evergreen ready” is not always easy, and it will most probably not happen over one night – but it is absolutely possible. Hopefully content of this article has helped proven that for you.

And remember: change is good!

MSOLDomain / MSOLFederatedDomain (Convert, Update, FederationProperty) Operations Fail with “Service not available”

BACKGROUND

Consider the following scenario:

  1. You are running, or are in the process of setting up, Office 365
  2. You are attempting to use Azure Active Directory Module for Windows PowerShell to do any of the following operations:
    1. Convert-MSOLDomainToFederated
    2. Convert-MSOLDomainToStandard
    3. Update-MSOLFederatedDomain
    4. Get-MsolFederationProperty
  3. The operation is aborted with reason “Service not available”. Extracts from example error message :

   + FullyQualifiedErrorId : InternalError,Microsoft.Online.Identity.Federation.Powershell.FederationPropertiesCommand

 Update-MSOLFederatedDomain : Service not available

CAUSE

This behaviour is caused by a bug in the 8362.1 version of the Azure Active Directory Module for Windows PowerShell (released January 19th 2015). When using this version, MSOLDomain/MSOLFederatedDomain operations fail with the “Service not available” error (the time of writing this is March 2015).

RESOLUTION

The resolve this, you need to uninstall the 8362.1 version of the Module, and revert back to the previous version – 8262.2. Follow these steps:

  1. Uninstall the Azure Active Direcotry Module for Windows PowerShell from the Control Panel (You can keep the required Sign-in Assistant installed).
  2. Visit this link: http://social.technet.microsoft.com/wiki/contents/articles/28552.microsoft-azure-active-directory-powershell-module-version-release-history.aspx#AO1
  3. Download version 8262.2 of the Module (direct link to 64-bit version here).
  4. Install the 8262.2 version, connect to AAD and attempt the operation again. This time it will work.

 

DirSync /fullsql Install-OnlineCoexistenceTool Configuration Fails (Error Code 1603, “FullyQualifiedErrorId : 5000″ and Event ID 1013)

BACKGROUND

Consider the following scenario:

  1. You are attempting to do a /fullsql installation of DirSync according to the instructions given in the MSDN article here. (/fullsql is needed only when more than 50 000 user, group and/or contact objects will be in scope for synchronization from the local Windows Server AD to the Azure AD/Office 365).
  2. The SQL server you are attempting to use is running Microsoft SQL Server 2008 SP1 or higher.
  3. The dirsync.exe /fullsql installation completes without any errors.
  4. The next step of running Install-OnlineCoexistenceTool with all SQL parameters (UseSQLServer etc) fails with the following error:

Install-OnlineCoexistenceTool : The Synchronization Engine installation
returned error code 1603. Please try the installation again, and if this error
persists, contact Technical Support.

Verbose output also gives:

Install-OnlineCoexistenceTool : The Synchronization Engine install returned
FAIL. See the event logs for more detailed information.
At line:1 char:1
+ Install-OnlineCoexistenceTool -UseSQLServer -SqlServer
    + CategoryInfo          : NotSpecified: (Microsoft.Onlin…CoexistenceTool
:
InstallOnlineCoexistenceTool) [Install-OnlineCoexistenceTool], DirectorySyncInstallException
    + FullyQualifiedErrorId : 5000,Microsoft.Online.Coexistence.PS.Install.InstallOnlineCoexistenceTool

Install-OnlineCoexistenceTool : The install was unable to setup a required
component. Check the event logs for more information. Please try the
installation again, and if this error persists, contact Technical Support.

In Event Viewer on the server, you can find Error Event ID 1013:

Product: Forefront Identity Manager Synchronization Service — Forefront Identity Manager Synchronization Services requires a running instance of Microsoft SQL Server 2008 SP1 or better. Install the correct SQL Server version and make sure the service is running before installing Forefront Identity Manager Synchronization Service.Event 1013

CAUSE

Taken that the SQL Server actually is running the said version or above, this error indicates that the SQL Native Client is missing on the server. (This requirement is now documented in the MSDN article, but is often missed as part of the DirSync preparation steps).

RESOLUTION

To resolve this error, download and install the SQL native client on the DirSync server.

Follow these steps:

  1. Visit this Microsoft Download Center and find the SQL Server Feature Pack for the version of SQL that you are running (Search for SQL Server version Feature Pack). If you are not sure what exact version of SQL Server you are running, you can always use this link (it is compatible with all versions of SQL that are compatible with DirSync): http://www.microsoft.com/en-us/download/details.aspx?id=44277
  2. Hit the Download link and select this file: ENU\x64\sqlncli.msi (SQL Native Client).
  3. Run the sqlncli.msi installer on the server to install the SQL Native Client.
  4. When installed, run the Install-OnlineCoexistenceTool commands again with your appropriate parameters and the installation/configuration will complete successfully.

REFLECTION

The SQL Native Client requirement for DirSync /fullsql is documented on MSDN nowadays, but I still often hear about enterprise DirSync deployments that get’s stuck on this issue – perhaps because the link provided in the MSDN article leads to general information about the SQL Native Client and not an actual download link to the feature needed on the server itself. In AADSync (next generation DirSync), the SQL Native Client is always installed as part of the AADSync installation, but that is not the case with DirSync. As DirSync will be around until the GA of AAD Connect (which will include AADSync), I thought it would be a good idea to post this until then – DirSync is and will be the generally preferred directory synchronization tool as long as it is the tool you get when following the Download link in the Setup Directory Synchronization section of the Office 365/AAD Admin Portal.

Solving Problems Accessing the Microsoft Intune Admin Console on IE11 (Silverlight and ActiveX Filtering Errors)

BACKGROUND

Consider the following scenario:

  1. You have successfully activated EMS or Microsoft Intune license SKUs to your Azure AD Subscription and to your admin account.
  2. You can successfully access and use the Microsoft Intune Account Portal (https://account.manage.microsoft.com/)
  3. When attempting to access the Microsoft Intune Admin Console (https://admin.manage.microsoft.com/) using your admin account and IE11, the Dashboard is never presented – instead you get stuck on a blue spinning circle that never disappears (see picture 1), or you get a message saying “This application requires Microsoft Silverlight.” even if Silverlight is already installed on the machine (see picture 2).
  4. When getting the second symptom, a link is provided to “Get Microsoft Silverlight”. When attempting to install Silverlight from the provided link, you get a message saying that “The same version of Silverlight is already installed”.

Spinning circle

This application requires Microsoft Silverlightintunesl3

CAUSE

The first symptom (the spinning blue circle that never disappears) may be related to ActiveX Filtering. Disabling it ( Gear Icon or Tools / Safety / ActiveX Filtering) should make the Dashboard load – or present the second symptom.

The second symptom (Dashboard does not load but instead say that the machine is missing Microsoft Silverlight even if it is installed) is related to a faulty procedure to detect Silverlight within Microsoft Intune.

The third symptom (Silverlight installation fails if you follow the link to download and install Silverlight from the message presented as part of symptom #2) is expected, as there is not a problem with your Silverlight installation – the message appears due to an issue related to Microsoft Intune and how it detects Silverlight on the machine.

RESOLUTION

Follow these steps to be able to access the Microsoft Intune Admin Console using IE11:

  1. Open Internet Explorer 11 and navigate to https://admin.manage.microsoft.com/
  2. If you get Symptom 1 (Spinning Circle), click the Gear Icon and click Safety \ ActiveX Filtering. Reload the page. This should either load the Dashboard or produce Symptom 2.
  3. If you get Symptom 2 (Message that Silverlight is not installed on the machine when it actually is), close all instances of Internet Explorer and navigate to Windows Control Panel \ Programs and Features and uninstall Microsoft Silverlight.
  4. After the uninstallation of  Microsoft Silverlight finishes, open Internet Explorer and navigate back to https://admin.manage.microsoft.com/. This time, follow the link “Get Microsoft Silverlight”, download and run the linked file and complete the installation wizard.
  5. Close all instances of Internet Explorer and start it again, navigate back to https://admin.manage.microsoft.com/. The Dashboard should now load, but if it still does not and Symptom 1 or Symptom 2 appears even this time, click the Gear Icon and click Safety \ ActiveX Filtering, then reload the page. The Dashboard will load.

Microsoft Intune Admin Console Dashboard

Hands on with AADSync (RTM) / AAD Connect – a Guide to Multi-Forest AD Synchronization and Attribute Filtering

ARTICLE UPDATED JUNE 2015 TO INCLUDE AAD CONNECT (NEW NAME, VERSION AND BUNDLING FOR AADSYNC).

INTRODUCTION

Office 365 is in many aspects best compatible with an on-premise infrastructure where the customer have followed Microsoft’s guidelines and best practices for design and configuration. A perfect example of such guideline is to keep all Active Directory user data associated with employees (personal user objects) within a single Active Directory forest. For many organizations, this is not the reality, often due to a background of growth by acquisitions where the acquired organisation remain their Active Directory data in their own forest that they always have been using. Such landscape is referred to as a “Multi-forest scenario” (one organization with users are spread across many Active Directory forests).

The aspect that makes the multi-forest scenario problematic is the user provisioning and administration. For single-forest customers (the number of domains within the forest does not matter), it have always been easy to deploy Directory Synchronization from your Windows Server AD forest to Azure AD by using DirSync – but that method have not been supporting the multi-forest scenario (DirSync supports single-forest synchronization only). Different alternatives have since the launch of Office 365 become available to address directory synchronization in the Multi-forest scenario (like the FIM AAD Connector), and there are still different ways to go at it – but now, better native support for this scenario is becoming a reality with the new Azure Active Directory Connect (AAD Connect) that previously was available as AAD Sync (AAD Connect is the new name, version and bundling for AAD Sync). This article will provide an overview of the alternative methods, and a detailed step-by-step guide on how to configure multi-forest synchronization from two separate Active Directory forests to one single Azure AD/Office 365 tenant. We will also look in to the feature of Attribute Filtering within AAD Connect to demonstrate how selected AD attributes can be skipped in the synchronization (please note that filtering out attributes is now possible but not generally recommended, I will explain why later in the article).

MASTERING THE MULTI-FOREST SCENARIO – OPTIONS OVERVIEW (OR “READ THIS FIRST”)

If you are considering Office 365 (or other AAD integrated applications) and multiple Active Directory forests is your reality, consider the options below. Option D will later in the article be explained in detail. There is no preferred order or priority for the given options.

OPTION A – CONSOLIDATE THE FORESTS FIRST: If Active Directory consolidation (merging the forests in to one) is an option, that activity could be completed prior to adopting Azure AD/Office 365 to avoid the multi-forest scenario complexity. The objects that needs to be consolidated to one forest are the user objects for the employees and groups that relate to any of the Office 365 services (i.e distribution lists and mailbox security groups). Computer objects and service accounts etc can still remain in the trusted forest when activating AAD/Office 365 services.

OPTION B – MIX SYNCED IDs WITH CLOUD IDs: If the majority of users resides in one AD forest, and a limited number of users are in a separate forests, a good option can be to implement AD integration (AADSync, ADFS) to the primary forest, but create Cloud ID (separate identities in Azure AD) for the users that are in the separate forests. This creates a mix of integrated and separate identities in the same tenant. If you are using ADFS in the primary forest, keep in mind that users in the other forests (that get’s the Cloud IDs) can not have the same domain in their username, because federation is activated per domain (if domain1.com is activated for federation, domain1.com users can not use Cloud IDs with user@domain1.com usernames, they would have to use another domain for their usernames).

OPTION C – USE SHADOW ACCOUNTS: This option is similar to Option B, but with the ability to centrally manage all identities. Instead of creating Cloud IDs for the users residing in the separate forest(s), create corresponding user objects (“shadow accounts”) in the primary AD forest for those users instead, and let AAD Connect provision these accounts in AAD/Office 365. These “shadow accounts” allows the users in the separate forest(s) to login with a centrally managed account, but they will not have Single Sign-on with ADFS as they are not logging in to Windows with the same account as they are using for AAD/Office 365. This option allows ADFS Claim rules, centrally managed password policies and other benefits that comes with proper Active Directory integration – but have considerations regarding end-user experience for the shadow users (there is a separate password associated with the shadow account, etc).

OPTION D – SYNCHRONIZE ALL FORESTS WITH AAD CONNECT: The final option is to actually enable multi-forest synchronization. This can be accomplished either by using the AAD Connector in FIM that still is available (but should not be used as it will not be developed further), or by using AAD Connect. It is recommended to use AAD Connect, as the FIM agent (as well as DirSync) is believed to be deprecated over time and replaced with AADSync completely. AAD Connect works like DirSync always have, but have native support for connecting with multiple Active Directory forests. How to implement and configure this option is explained in detail below.

IMPLEMENTING MULTI-FOREST SYNC WITH AAD CONNECT – SCENARIO OVERVIEW

I will now explain how to configure multi-forest synchronization step by step by step. The screen shots are collected from AADSync, but from June 25 2015, you should downlad and use AAD Connect to configure multi-forest – the wizard in the screenshots matches the one in AAD Connect for the AAD Sync steps. Before jumping in to configuration, let’s have a look at the given scenario.

SOURCE: Two separate AD Forests containing users – FOREST-A.int and FOREST-B.int
TARGET: One Azure AD/Office 365 tenant.
METHOD: We will configure AAD Connect (AAD Sync used in the screen shots) to synchronize objects from both forests to the tenant. We will also configure AADSync to filter out the ipPhone, homePhone and extensionAttribute9 AD attributes from the synchronization.

Overview. Users from FOREST-A and FOREST-B will be syncrhonized to Azure AD.

Overview of users in FOREST-A

Overview of users in FOREST-A

Overview of users in FOREST-B.

Overview of users in FOREST-B.

CONFIGURING MULTI-FOREST DIRECTORY SYNCHRONIZATION STEP-BY-STEP

Pre-requisites:

  1. You have an AAD/Office 365 tenant
  2. You have activated all your smtp-domains that are used in the organization (both forests) in your tenant.
  3. Your users have either or both their UPN-name and mail-attribute in Active Directory populated to match their primary smtp-address.
  4. You have one machine available to install AADSync (can be joined to any of the forests). The machine can access domain controllers in both forests and is able to resolve both forest DNS names.
  5. You have user account credentials to read both AD Forests, and you have a Global Administrator account to write to your Azure AD.
  6. The total number of objects that you will syncrhonize is not more than 100 000 (would require a full SQL setup which is not covered in this guide). The limit for full SQL setup was 50 000 in DirSync, but is now lifted to 100 000 for AAD Connect.

Make sure you meet all pre-requisites above. Then, let’s get started.

  1. The first step is to activate Active Directory Synchronization in your tenant. Do so by logging in to the admin portal, browse to users and follow the link at the top for setting up AD Synchronization.

    1activatedirsync

    Navigate to Users in your admin portal and follow the highlighted link to set up AD Synchronization. Notice in the screenshot that there are only two users (Cloud IDs) present in the tenant as of now.

  2. Click Activate to enable AD Synchronization in your tenant.
    2activatedirsync
  3. Download AAD Connect to the server where you want to install the tool. Get it here.
  4. Launch the installer executable (MicrosoftAzureADConnectionTool.exe).
    5launch_aads
  5. The installer let’s you choose the installation path and accept license terms. Click Install to continue.
    6wizard_Start
  6. The necessary components are now installed to the server (Sign-in Assistant, SQL LocalDB, SQL Native Client, the actual Sync Service etc). After the component installation, the wizard interface may disappear completely for about 30 seconds, but it will come back when it have finished loading the configuration wizard.
  7. The first step in the configuration wizard is to connect to your Azure AD. Enter the credentials for a Global Administrator in the tenant and hit Next to continue.
    7wizard1
  8. Next, enter credentials for the first forest you want to synchronize. Start by adding the forest to which you have joined the AADSync server (in this case this is FOREST-A). Hit Add Forest to verify the connectivity and credentials, and to add the next forest.
    8wizard2
  9. Add the next forest. Notice that I use the forest FQDN to specify the domain in the Username field (FOREST-B.int\jesper). I recommend this when adding the external forests to avoid an error that states “The specified domain does not exist or cannot be contacted.”, which may occur if only using the NetBIOS name (FOREST-B\jesper).
    9wizard3
  10. When both forests are added to the list, hit Next to continue.
    10wizard4
  11. The next option screen in the wizard covers settings for user matching. These options require careful consideration if you are using GAL Sync (or for any other reason have users represented as User and/or Contact objects in both forests) and/or are planning for a future Active Directory consolidation/migration of the forests that you currently are configuring for AADSync. Read the following rationales for these options before proceeding:


    Matching across forests: If your user objects only occurs in their respective forest, and are not replicated as a contact object or duplicate account in the other forest, leave the “Matching across forests” option at its default – “Your users are only represented once across all forests”. If objects are represented in many forests, you must make a selection for an attributed that can be used as a matching attribute to merge/join the duplicates to one Azure AD identity, or you will have collisions during the synchronization. More information is available if you follow the “Learn more about user matching” link in the wizard.

    Matching with Azure AD
    : These two options are used for identity federation. The sourceAnchor attribute is the immutable ID for the user, and must not be changed during the lifetime of a user object. The default choise – objectGUID – is a good choise IF YOU ARE NOT PLANNING AN ACTIVE DIRECTORY CONSOLIDATION OR MIGRATION IN THE FUTURE. The objectGUID attribute will change if the user is moved to another forest, and would in that case create a duplicate user in Azure AD (and a big mess to clean up). Use the objectGUID if you are certain that the affected user objects will remain in their current forest for their remaining lifetime. The userPrincipalName attribute is the attribute that will populate the Username in Azure AD (the corresponding UPN-name in AAD). Having users signing in to AAD/Office 365 with their primary smtp-address as their username is a best practice, so use the default – userPrincipalName – if attribute is matching the users smtp (email) addresses (or if you are planning them to do so). If your UPN names do not match the primary smtp-address and you are not planning on changing the UPNs to do so, consider using the mail-attribute instead (in that case, just remember to configure ADFS to use the same attribute by following these instructions).


    In summary: If your users are represented once across the forests (no GAL Sync or equal in place) and if your UPN-names are matching primary smtp-addresses, you can go ahead with the default options here, like I do in this example. Make your selection and hit Next to continue.
    11wizard5
  12. Next up are Optional features. At the time taking this screenshot, Password Synchronization was not available in AADSync (RTM version, build 1.0.0419.0911), but this is available now (from October 2014). Password write-back is also in its final stages of development and will soon be available to customers that have an Azure AD Premium subscription (not included in Office 365). The Exchange Hybrid option must be selected if you are planning to set up an Exchange Hybrid Configuration between one of your on-premise Exchange organizations and Exchange Online (Multi-org Exchange hybrids are also in development but not yet available at the time writing). I am selecting Exchange hybrid deployment and Azure AD app and attribute filtering to demonstrate the new capabilities to filter out attributes from AADSync. Please note that I do not recommend that you use app and attribute filtering (see explanation below under “Should I filter out apps…”), I am making the selection in this example to demonstrate the capability. Make your selection and hit Next to continue.
    11wizard6
  13. If you selected the “Azure AD app and attribute filtering” in the previous step (not generally recommended), you will now have the option to filter out Azure AD apps (Office 365 services etc). Filtering out apps and attributes is possible, but before you decide doing so, please read the following rationale:
    Should I filter out apps and/or attributes from the synchronization? My general answer to this question is “No”. There are in my opinion three main reasons why you shouldn’t.REASON A – NO SECRETS IN ACTIVE DIRECTORY: By default, all users can read all attributes (not the passwords) for all objects in Active Directory. If you want to filter out attributes because you have secret data stored in AD attributes, you are doing things very wrong. Any user can open any AD or LDAP administration tool (like Active Directory Users and Computers) and read and/or export all your directory data from any machine in the network. If you are allowing shared AD accounts (like POS/kiosk accounts), you will not have any chance to log or trace such export either. You should not store any secret data in Active Directory attributes (another Microsoft Best Practice), and therefore synchronizing attributes to AAD that any user can read anyway should not be imposing a security threat to your organization.     
    REASON B – KEEP IT SIMPLE, IT IS YOUR PRIVATE DATA: If you filter out apps/attributes in your synchronization, you are immediately configuring a “special setup” with a complexity parameter added to it. Your tenant and your Azure AD is your private data. You are not sharing it with Microsoft or anyone else. Keep your AAD synchronization solution as simple as possible, trust Office 365, and synchronize all attributes as recommended to avoid incompatibility (surprises) in the future as AAD/Office 365 develops.     
    REASON C – FUNCTIONALITY AND RICH USER EXPERIENCE:
    To allow the very best end-user experience, GlobalAddress Lists, Lync Contact Cards, SharePoint profile sites etc should all be populated with as much information possible. If you decide to filter out attributes, you will take away functionality and “rich content” from your end users. It is also worth mentioning that Microsoft services, such as Exchange and Lync, depend on Active Directory attributes for their functionality – so filtering out an attribute that is not technically needed today might cause an upcoming feature not to work in your environment in the future. In this example, we are filtering out three attributes, but we will leave attribute scopes for all apps intact in this option screen. Make your selection and hit Next to continue.
    12wizard7
  14. As we selected to use “Azure AD app and attribute filtering” previously, we now get the chance to filter out specific attributes (the previous options screen for apps let us filter out sets of attributes based on specific apps/services). To demonstrate the attribute filtering capability, I am here filtering out the ipPhone, homePhone and extensionAttribute9 attributes from synchronization because we don’t want them to be visible in the Exchange Online Global Address List. I make the selection by first checking the “I want to further limit…” check box and then clearing the attributes from the list. Make your selection and hit Next to continue.
    13wizard8
  15. Next is a summary of your AADSync configuration. Check to see that everything looks OK and hit Configure to continue.
    14wizard9
  16. When the configuration is completed, you have the option to start the first Synchronization immediately, or to finish the wizard without starting the synchronization so that you can configure filtering before starting (if your would like to filter out specific OUs, child-domains or set an attribute based filtering). Filtering with AADSync works and is configured the same way as it always have with DirSync. Instructions on how to enable and configure filtering can be found here. In this example, we want to synchronize the complete scope of both forests, so we let the “Synchronize now” option stay selected and hit Finish to complete the wizard and start the synchronization.
    15wizard10
  17. Now, all we need to do is wait for the synchronization to complete. You can monitor the synchronization progress by opening the Synchronization Service Manager (the miisclient interface) from the Start menu. You can track the different stages of the synchronization in the Operations tab.
    16SyncServiceManager
  18. There is also another tool available with AAD Connect called the Synchronization Rules Editor (you will find it in the Start menu). This tool can be used to see and edit the attribute flows and scoping filters. You can use the tool to create transformation rules for certain attributes etc. In this example, we will not make any such changes, but we open it to have a look while we are waiting for the synchronization to complete.
    17SyncRuleEditor
  19. When the synchronization is completed, it is time to see the result. Sign in to the Office 365 admin portal and navigate to Users. Now, you will see users from both forests provisioned as synchronized identities “Status is Synchronized with Active Directory”. Success!
    18Syncedusers
  20. When opening a user from each forest, you see that both users are synchronized (get’s the banner stating they can only be edited in the local Active Directory) and that the user name matches the UPN-name in the local AD.
    19ForestAuser20ForestBuser
  21. The users can now be licensed and start using Office 365, or other AAD-integrated, services. You may also set up federated identities using ADFS and provide Single Sign-on to the users from both forests using a shared ADFS STS service with proper configuration.

SUMMARY AND DISCLAIMER

In this article we have seen how powerful and yet easy to use the new AAD Connect Service is. Multi-forest is a common and challenging reality for many enterprise customers, and with AAD Connect there is finally a robust, simple and supported way to get Active Directory synchronization working in that scenario. So thank you DirSync, FIM AAD Connector and stand-alone AADSync for the great time together, it have been great – but now it’s time for AAD Connect to take it from here.

ADFS 3.0 and Hardware Load Balancing – the Lost Documentation

Article updated October 2014 to include http based health check probe as a workaround, made available with August 2014 Update Rollup for Windows Server 2012 R2.

BACKGROUND AND PURPOSE

ADFS and Federated Identities are common components for Enterprise Office 365 customers necessary to allow Single Sign-on, use of Claim Rules and immediate account lock-outs etc. As the availability the ADFS service decides the availability of Office 365 (if you can’t authenticate you can’t use the service), load balancing is a must-have. In most cases, Hardware Load Balancing (HLB) is used, and for ADFS 2.0 (Windows Server 2008/2008R2) and ADFS 2.1 (Windows Server 2012) this have never been a problem. You deployed your ADFS STS and/or ADFS Proxy, and published them with your Hardware Load Balancer like “any other SSL site” and it just worked.

Today, ADFS have with Windows Server 2012R2 reached version ADFS 3.0 – and the full “out-of-the-box-support” for using “any Hardware Load Balancer” is not as obvious any more. Making it  work is a bit more tricky, depending on what HLB you are using. The purpose of this article is to present my lessons learned from deploying ADFS 3.0 with HLB – and to save you from hours of trouble shooting when getting your own ADFS 3.0 HLB up and running with Office 365.

ADFS 3.0 AND HARDWARE LOAD BALANCING – WHAT COULD POSSIBLY GO WRONG?

ADFS 3.0 and Web Application Proxy (WAP) in Windows Server 2012R2 uses an extension to the TLS SSL protocol called Server Name Indication – SNI (RFC 4366). This extension allows web servers to present host names when handshaking SSL, so that multiple SSL sites can be hosted on a shared IP-address and port (443) – just like the concept of host headers in good old http have always allowed us to do. This is great because it allows preservation of IP-addresses which also can reduce a total cost of ownership for a service (public IP-addresses does not come for free these days).

What is not so great is that the SNI extension is not yet supported by “everything else in the world” – such as very old web browsers – and Hardware Load Balancers. Using a HLB that does not have a clue about SNI will not work, and that is the topic of this blog post.

If you are interested to read more about SNI in Windows Server 2012R2 – and how to solve issues that can occur, I can recommend Ian Parramore’s two excellent blog post on TechNet, which also have been a great inspiration and source of content to this post, found here, and here.

HOW DO I KNOW THAT MY HLB DOES NOT SUPPORT SNI?

If your HLB does not support SNI, you will notice it eventually when attempting to configure a load balancing and you don’t get any response on port 443 from any of the server IP-addresses. (that’s usually when you spend a day or two troubleshooting firewall ACL’s and routing without any luck). My best advice is to look for SNI-related settings in the HLB administrative interface and read through the release notes for the software version you are running on the appliance(s).

Examples of observed behaviour when attempting to configure HLB with a non-SNI compliant device is a “Http/1.1 Service Unavailable” message from a Citrix NetScaler device and a “Network Error (tcp_error) A communication error occured: Connection refused” message from a BlueCoat device.

Big-IP/F5 customers may turn to a technical article written by Greg Coward for assistance in configuring the Big-IP to work with ADFS 3.0/WAP, found here.

MY HLB DOES NOT SUPPORT SNI – WHAT CAN I DO?

CHECK FOR UPDATES: Before attempting to fix the issue on the Windows Server side – you should always see if there is an available software/firmware update for your HLB device that includes SNI support. There are two workarounds available (see below), but having native support is usually preferred as it reduces the overall complexity of your solution.

WORKAROUND #1 (RECOMMENDED): ADD A HTTPS (PORT 80) BASED HEALTH CHECK PROBE

NOTE: This functionality was made available with the August 2014 Update Rollup for Windows Server 2012R2, found here. Running at least this update rollup level on your ADFS/WAP servers is required for the following workaround to work.

By adding a http-binding on the ADFS/WAP server, your HLB can do the health check probing using the http protocol and thus bypassing SSL and SNI completely. The http endpoint will respond with HTTP 200 OK if available if queried to http://servername-or-ip-address/adfs/probe, and will not automatically allow your ADFS traffic to use HTTP, they will function just as health check probes. Follow these steps to create such binding:

  1. Make sure you have installed all available Windows Updates on the ADFS/WAP machine – at least August 2014 Rollup.
  2. Launch the Windows Firewall with Advanced Security MMC on the first WAP server
  3. Go to Inbound Rules
  4. Create a new Inbound Rule (New Rule in Action Pane). Suggested name: “ADFS HTTP Health Check Probe”
  5. Configure the rule for TCP protocol, local port 80 (specific port) and Allow traffic (All ports as Remote port). See overview of expected result in this picture from Ian Parramore’s blog.
  6. Repeat steps on other ADFS/WAP machines.
  7. Configure HLB to do health check probe using the http protocol and this specific endpoint: http://servername-or-ip-address/adfs/probe

WORKAROUND #2: ADD A FALLBACK BINDING TO 0.0.0.0:443

It is possible to create a binding to make ADFS 3.0 and/or WAP to respond to SSL requests made out to ip-adresses. This is the recommended workaround as you can’t fully disable the SNI functionality completely. The process of adding this fallback binding consist of finding out the certificate thumbprint of your desired SSL certificate (added with private key to the Local Computer\Personal store) and then binding that thumbprint to all ip-adresses and for the ADFS or WAP application.

Follow these steps on all your ADFS 3.0 servers to add the fallback binding (and make your non-SNI compliant HLB be able to see your ADFS servers):

  1. Make sure that you have installed all available updates for Windows Server 2012R2 after adding and configured the ADFS STS or WAP Proxy role.
  2. Open a Command Prompt as administrator
  3. Run the following command:
    netsh http show sslcert
  4. You will see a list of SSL Certificate bindings.
  5. Mark and copy the ‘Certificate Hash’ value. Paste it to a temporary place (i.e Notepad)
  6. Mark and copy the ‘Application ID’ value. Paste it to the temporary place. The Application ID is what will associate the binding with ADFS 3.0 (for the internal STS servers) and WAP (for the ADFS Proxy).
  7. Now run the following commmand, where you insert the noted ‘Certificate Hash’ and ‘Application ID’ values from above (keep the { } characters):
    netsh http add sslcert ipport=0.0.0.0:443 certhash=Insert_Certificate_Hash_Here appid={Insert_Application_ID_here}
  8. Restart machine and repeat steps for remaining ADFS 3.0 machines.

After adding the fallback binding, your HLB should be able to reach your ADFS servers normally, and you can continue to configure and fine tune your load balancing. Be aware that if you replace your certificate in the future (when it expires), you must perform these steps again (a new certificate means a new certificate hash) – worth mentioning in your system documentation.

SUMMARY AND DISCLAIMER

Adding a http health check probe or a fallback binding can help us fix our load balancing, but my hope is that SNI support in HLB devices will become more of a standard than it is today. Like stated in the article, looking for native support with a software/firmware update and proper documentation from your HLB vendor is always recommended before applying the fallback binding workaround. If you are aware of SNI-related knowledge base articles and software updates from HLB vendors, please share them with me in the comments field below or on Twitter (@JesperStahle) and I will add them to this article.

Internet Explorer 8 Only Loads Outlook Web App Light in Exchange Online/Office 365

BACKGROUND

Consider the following scenario:

You are using Exchange Online/Office 365. When opening Outlook Web App (OWA), Internet Explorer only loads the Light version of OWA (OWA Light). The same user on the same client could earlier load the rich OWA experience even on Internet Explorer 8.

If the same user logs on to Internet Explorer 9 or later, or the latest version of Firefox, Chrome or Safari, rich OWA is loaded instead. OWA Light is only enforced on Internet Explorer 8.

CAUSE

This behavior is by design. Rich OWA have earlier been allowed for Internet Explorer 8 users, but with limited performance (see this article). Starting May 2014, OWA Light is being enforced for Internet Explorer 8 users. This is stated in the Office 365 Service Description, see this article.

Quote from the Service Description:

“The user experience sending and receiving email with Outlook Web App and Internet Explorer 8 might be substantially diminished, especially when used on Windows XP or with low memory devices. Office 365 does not offer code fixes to resolve problems you encounter when using the service with Internet Explorer 8, and new Office 365 experiences might not work at all. You should also expect the quality of the user experience with Internet Explorer 8 to diminish further in the near future. Beginning in May 2014, Internet Explorer 8 will only display Outlook Web App Light.”

RESOLUTION

Upgrade Internet Explorer to the latest version, or use the latest version of an alternative browser. If you have legacy application dependencies to Internet Explorer 8, consider upgrading to Internet Explorer 11 and use IE Enterprise Mode to emulate IE8 rendering for those legacy web sites/applications.

And remember – Office 365 is an “evergreen” service, meaning that back-end updates will be released constantly. As a customer, you should keep your client applications on the latest version/build to stay fully compatible and supported. This includes Windows, Office applications and Internet Explorer.

Lync Client Asks for Credentials (“EWS not Deployed”) in an Exchange Hybrid Configuration

BACKGROUND

Consider the following scenario:

  1. You have successfully established an Exchange Hybrid Configuration between your on-premise Exchange Server installation and Exchange Online.
  2. You are using a fully functional Lync Server on-premise installation for all users.
  3. DNS zones for any of your primary SMTP domains exist in the internal network DNS (split-DNS is used)
  4. Some users are migrated to Exchange Online, and some are still residing in the Exchange Server installation on-premise.

The following symptoms occur:

  1. For users that are migrated to Exchange Online, sporadic credential prompts appear from the Lync client, asking users for username and password. Users that still have their mailbox residing in Exchange Server on-premise are not asked for credentials.
  2. The following message is displayed in Lync Configuration Information (right-click the Lync icon in the System Tray to find this option): EWS Information: EWS not Deployed

CAUSE

Two ways to update presence information: The Lync client is by default configured to update your presence based on your Exchange calendar (Personal information manager settings in the Lync client). Lync can read your presence information from Exchange by either receiving it through the Exchange Web Services (EWS) API, or by reading it from Outlook (if Outlook is installed and running). The EWS method is dependant on Exchange Autodiscover to be fully functional (details will follow under Resolution).

If, as in the scenario explained above, the Configuration Information indicates “EWS not deployed”, the Lync client will fail to use the EWS API to update presence information. Instead it will fall back to updating the presence information from Outlook.

But it worked before: The Outlook-method have so far worked fine, and will continue to work fine, for non-migrated users as the Lync client is running under the same user context as Outlook. When users are migrated to Exchange Online and configure Outlook to run in the Office 365 context (regardless if using ADFS/SSO or not), the Lync client will no longer be able to update the presence information from Outlook without first having you authenticate in the Lync client with the Office 365 credentials. Migrated users are no longer running Outlook and Lync in the same user context. This produces the credential prompt.

To resolve this, the EWS method should be used instead. The “EWS not deployed” error must be repaired.

RESOLUTION

Exchange Autodiscover is key: The Lync client will use Exchange Autodiscover to resolve the URL for EWS. Exchange Autodiscover is assumed to be working in this scenario as the Exchange Hybrid Configuration is established (which itself have a Heavy dependency to Autodiscover). If Exchange Autodiscover is a new topic for your, I suggest reading this article before continuing.

SCP vs. DNS: Outlook will use Active Directory Service Connection Points (SCP) to find the URL to Autodiscover on the internal network (Active Directory joined machines). Outside the internal network, Autodiscover URLs are distributed using public DNS (looking at the domain in your e-mail address and ask for Autodiscover records in that domain). As internal clients rely on SCP records in Active Directory, there have never been any need to add the DNS records for Autodiscover in the internal DNS zone (the split-zone) for the SMTP/SIP domains.

The Lync client will not use Active Directory SCP records to retrieve the Autodiscover URL. It will only use the DNS-record method. This differs from Outlook, which can use SCP records to find Autodiscover. Because of this difference, you might have a fully functional Outlook (finds Autodiscover via SCP-records) and a fully functional Exchange Hybrid Configuration (finds Autodiscover via A/SRV/CNAME records in the public DNS zone), but you will not have a functional Lync client (attempts to find Autodiscover from the internal split-zone for the SMTP/SIP domains).

How to fix it: In order for the Lync client to be able to find Autodiscover on the internal network, add A/SRV/CNAME records (whatever mirrors your public DNS records) in the internal DNS zone (the split-zone) to mirror the setup for Autodiscover records in the public DNS.

After adding internal DNS records for Autodiscover, the Lync client will be able to find Autodiscover and thus the URL for EWS. Through EWS, the Lync client can retrieve your presence information from the Exchange Online mailbox via the Exchange Hybrid Configuration, and users migrated to Exchange Online will no longer suffer from credential prompts appearing in the Lync client.

Office 365 and Web Proxy – the Lost Documentation

BACKGROUND AND PURPOSE

Running Office 365 together with web proxy is supported and also the reality for many (or most) global Enterprise customers. Moving to Office 365 with web proxy present in your landscape is related to a number of considerations and possible configuration steps in order to ensure the best possible performance and functionality.

Guidelines for customers using web proxy do exist in the Office 365 Service Description and Deployment Guide. However, the information is scattered between different articles, why it can be unclear for the solution architect or network administrator to get a clear view on the considerations and principles that apply when using web proxy with Office 365. The purpose of this article is to collect the available information (plus some notes from the field) in one place to form an assembly on the given topic. Note that the information in this article also applies to CRM Online.

THE CLOUD AND WEB PROXY – WHAT COULD POSSIBLY GO WRONG?

Having a web proxy and running cloud based services might seem like the perfect fit. “The service is on the Internet, and our web proxy accelerates the Internet traffic to our end users web browsers – what could possibly go wrong?”.

There is no general answer to what could go wrong – it depends on what Office 365 services you plan to enable, and also what type of web proxy you are using.

But, to give an idea, here are six examples of things that might break when deploying Office 365 behind web proxy without first taking the right precautions:

  1.  Office Pro Plus Click-2-Run installation fails with the reason: “Something went wrong – Sorry, we ran into a problem”, as the installer changes from winhttp to BITSAdmin at around 17% of the installation progress. (Explained here).
  2. Outlook might lose its Connection to the Exchange server intermittently. A “balloon message” in the lower right explains “The connection to Microsoft Exchange is unavailable. Outlook must be online or connected to complete this action.”
  3. Downloading files from SharePoint Online takes an unreasonable amount of time to complete.
  4. Send and receive in Outlook works, except for downloading the Offline Address Book (gives error 0X8004010F). Sporadic credential prompts may also appear.
  5. Outlook might take several minutes to start/load. (Explained here).
  6. If using ADFS with Office 365, the Microsoft Identity Platform Relying Party Trust throws an error stating “This relying party is out of date due to monitoring issues. Please check the event log for details”.

To remediate all the example errors above, and many others like them, read through and act upon the following considerations:

CONSIDERATION #1 – IF POSSIBLE, BYPASS THE WEB PROXY COMPLETELY FOR OFFICE 365

If your network configuration and security policies allows you to bypass your web proxy for selected services/sites on the Internet, it is strongly recommended that you consider adding Office 365 to such exception list and open your client networks for direct access to Office 365s URLs (strongly recommended) (using IP-ranges does not work in practice and is not recommended as per this article). Generally, Office 365 will always work best without the client being behind a web proxy (reasons explained in the categories below). If making such exceptions is not an option in your case and you will use your web proxy with Office 365, keep reading the following considerations and recommendations.

RECOMMENDATION: If excluding services/sites from the web proxy is an option, and you can allow web traffic directly from the client networks to selected URLs on the Internet, use this article to find the specifications for what URLs are used by Office 365 and allow the traffic directly  (using IP-ranges does not work in practice and is  not recommended). The list is subject to updates, monitor the Change Notification RSS feed to keep your access lists current. Also remember to add all URLs to the proxy bypass list on the client (in your PAC file or in the Internet Explorer settings) to avoid Office 365 traffic to go through the web proxy spitefully). The list of additional URLs needed for CRM Online is found in this article.

For the majority of the services (like using the portal, SharePoint Online, OneDrive for Business, Exchange Online and the Outlook client), you will only need to open port 80/443, but additional ports are needed for Skype for Business. Specifications on ports and protocols are found here.

CONSIDERATION #2 – NAT:ED IP ADDRESSES AND NUMBER OF CONCURRENT SESSIONS

This topic applies even without web-proxy, but definitely relates to your web proxy configuration.

In the Office 365 Service Description, there is a topic called “Plan for Internet bandwidth usage for Office 365″, found here. In the planning guidelines, a topic can be found called NAT support with Office 365.

In that topic, Microsoft explains the limitations that applies when using Network Address Translation (NAT) in respect to the amount of ports that are available to share between users behind the NAT:ed IP-address (public/Internet facing address). I recommend reading the article for details (sub-topic “NAT support with Office 365″), but here is the summary:

Clients for Office 365 in general, and Outlook in particular, opens sessions with the service in the cloud. These sessions consumes the shared amount of ports that are available per Internet facing NATed IP-address. The general guidance is to add a public facing IP address per 2000 concurrent users to share. Personally I recommend adding one NAT address per 5000 concurrent users.

RECOMMENDATION: Read the article above and scale out the amount of public IP-addresses to provide for your number of concurrent users. For example, if you are using one proxy server with one single NATed address to the Internet to provide for 10 000 concurrent users, add one or two more IP-addresses to you web proxy configuration (configure a NAT pool, add NIC or add additional IP addresses, whichever works for your web proxy). Add more IP-addresses for more concurrent users.

CONSIDERATION #3 – RFC1323 (TCP WINDOW SCALING)

An excellent article on this topic can be found here: http://blogs.technet.com/b/onthewire/archive/2014/03/28/ensuring-your-office-365-network-connection-isn-t-throttled-by-your-proxy.aspx

The article explains how disabling the TCP Window Scaling mechanism in any network device may cause very bad performance to Office 365 (and probably other services as well). In the example used in the article, the download time for a 14 MB PDF file is reduced from over 8 minutes to just 32 seconds by simply enabling the Auto Scaling mechanism in the web proxy. That is pretty remarkable.

RECOMMENDATION: Make sure that RFC1323/TCP Windows Scaling is enabled in all network segments from your clients to the Internet. See the TechNet-article for details. If you are running Bluecoat, you can follow this article to find out if you have RFC1323 enabled or not: https://kb.bluecoat.com/index?page=content&id=FAQ1006

CONSIDERATION #4 – NECESSARY EXCEPTIONS (BEWARE OF CACHING, TRAFFIC INSPECTION AND AUTHENTICATION PROMPTS)

On of the key benefits with having a web proxy is its ability to cache content from the web to increase performance and decrease use of Internet bandwidth. However, the Office 365 services is not always fully compatible with content caching, and it is not recommended to use such technology. Read this post for official statement and details: http://support.microsoft.com/kb/2690045/

The article is addressing WAN Optimizers, but the principal applies to web proxy as well. The bottom line is that caching the data might interrupt the service protocols and thus deteriorate performance rather than improving it.

Content caching will not improve any performance in Outlook as the user is working against a local copy of the mailbox (the OST-file created when Outlook is running in Cached mode). There is often a debate if content caching can improve SharePoint Online performance when working with large files – but my recommendation is to try without content caching and later enable it just for SharePoint Online to compare. If there is no, or a minimal, performance increase – you best stay without content caching to comply with the principles stated in the article above. For Skype for Business Online, content caching can not help increasing performance either.

Further, web proxies have the ability to inspect web traffic for malicious content and filter out suspicious traffic before it can be downloaded to any client. These technologies go under many names (Content Inspection, DPI, DCI etc), but the principal is the same – traffic is inspected in transport and rules may filter it if found suspicious. Such content inspection might also disturb Office 365 client protocols, and cause intermittent disconnection and/or performance decrease.

Another element of the web proxy is its ability to secure web traffic by only allowing authenticated users to access the Internet. This will require authentication of some sort for the user. If single Sign-on (SSO) authentication is enabled for the web proxy, this should not be a problem with Office 365, but if the web proxy presents credential prompts (operating system based or as forms in the actual browser) as part of the day-to-day user experience, you can expect those prompts to be a an issue for Office 365 to function as well – traffic should be allowed without authentication requirements.

RECOMMENDATION: Three parts are covered in this consideration and recommendation – content caching, traffic inspection and authentication without seamless SSO. None of these components are fully compatible with Office 365, why you should make sure to exclude Office 365 URLs in the web proxy for all of them. The web proxy will function just as a proxy/relay for the Internet traffic, but will not cache its content, inspect its traffic or require authentication if the user have not already authenticated. The URLs to exclude can be found in this article. After adding the exception list in your web proxy, make sure to visit the Change log for URLs now and then to check for updates. There is also an RSS feed of the updates that you can subscribe to. The list of additional URLs needed for CRM Online is found in this article.

CONSIDERATION #5 – PAC SCRIPTS AND IE SETTINGS

There are different methods of distributing the web proxy settings to the clients in the network. One is to Proxy Auto-Config (PAC) files that you either distribute to your browser automatically with the WPAD protocol (DHCP and DNS lookups) or you can specify the URL to the PAC manually or by using Group Policies (just make sure to never use file:// paths). If not using PAC at all, you can specify the hostname directly to the web proxy, also manually or via Group Policy.

All methods works with Office 365, but problems may occur if you are not using PAC files, but your Internet Explorer settings suggest that you do (“Automatically detect settings” or “Use automatic configuration script” is enabled).

The first symptom of this is usually that Outlook takes several minutes to load (explained here). Another symptom might be that 3rd party applications that attempt to access Office 365 services (such as a mailbox in Exchange Online), simply can’t connect to the service. The Internet Explorer settings are very sensitive, and they might require some adjustments before everything operates as expected. Often, the setting “Bypass proxy server for local addresses” plays an important role in the trouble shooting.

RECOMMENDATION: Generally, the most solid setup for Office 365 together with web proxy is when using PAC files with the correct settings in Internet Explorer. If you are not using PAC files, make sure that your IE settings do not suggest that such file exist on the network, and keep your eyes and ears open for bad Outlook performance and similar indications.

SUMMARY AND DISCLAIMER

As stated initially, the first recommendation is to exclude Office 365 from the web proxy completely and allow clients to reach the specific URLs directly from the client. If such exception is not an option, the other considerations and recommendations in this article will hopefully help you to achieve the best possible performance and experience with Office 365 (and/or CRM Online).

Please note that this article is written based on field experience, and that the scenarios explained might or might not apply to your scenario and your specific web proxy. Any feedback to this article is very welcome to my Twitter, @JesperStahle. My intention is to keep this article up to date with guidelines for using web proxy with Office 365 – so if you know more considerations and recommendations that you should make the list, please let me know.

Hands on with Office 365 Message Encryption (OME) using One Time Passcode

ARTICLE UPDATED OCTOBER 2014 AFTER LAUNCH OF OTP CAPABILITIES IN OME.

INTRODUCTION

At the beginning of 2014, Microsoft released new capabilities for email encryption, called Office 365 Message Encryption (OME). The functionality was first announced in november 2013 in this article. In October 2014, OME was improved drastically as the receiver of encrypted emails now can use a One Time Passcode (OTP) to access the encrypted message (documented here). At launch of OME, all receivers had to authenticate with either an Organizational Account (Azure AD) or a Microsoft Account (formerly LiveID) to access the email, which often led to confusion for external receivers. Now, the receiver can instead request a One Time Passcode to the same email address as to which the encrypted message was sent, and use that (within 15 minutes) to access the encrypted email. This post was updated in October 2014 to include this new OTP-functionality.

In this post, I will demonstrate the user experience of Office 365 Message Encryption, both for the end-user (using OTP) and the administrator (setting up IRM Licensing and all necessary Transport Rules etc).

SCENARIO

We will create a transport rule that will enable Office 365 Message Encryption on messages with a Sensitivity level set to Confidential.

STEP 1 – ENABLE IRM LICENSING

If you attempt to use Office 365 Message Encryption before first enabling IRM licensing, the operation will fail and give you this message:

You can’t create a rule containing the ApplyOME or RemoveOME action because IRM licensing is disabled.

To remediate this, we need to enable IRM licensing using the Admin portal and PowerShell. Follow these steps:

  1. First, enable RMS in your tenant by logging on to your Office 365 Admin Portal, navigate to Service settings (left pane). select Rights management (top bar), click Manage and finally hit the button Activate. A message should appear stating that RMS is activated for the tenant.
  2. The remaining steps will be done in PowerShell. Open Windows Azure Active Directory Module for Windows PowerShell, found here:
    http://technet.microsoft.com/library/jj151815.aspx
  3. Connect your PowerShell session to your Office 365 tenant an Exchange Online by entering the following:
    $msolcred = Get-Credential (Answer the credential prompt with your Office 365 tenant administrator credentials) 
    Connect-MsolService -Credential $msolcred

    $Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $msolcred -Authentication Basic -AllowRedirection
    Import-PSSession $Session
  4. When connected to Exchange Online, you can enable IRM licensing with just a few steps. The first step is to set the RMS Online key sharing location. You will use different configurations depending on where your tenant is located (North America, European Union or the Asia-Pacific area).Enter the command that matches your tenant location (choose one):
    North America: Set-IRMConfiguration -RMSOnlineKeySharingLocation https://sp-rms.na.aadrm.com/TenantManagement/ServicePartner.svc
    European Union: Set-IRMConfiguration -RMSOnlineKeySharingLocation https://sp-rms.eu.aadrm.com/TenantManagement/ServicePartner.svc
    The Asia-Pacific Area: Set-IRMConfiguration -RMSOnlineKeySharingLocation https://sp-rms.ap.aadrm.com/TenantManagement/ServicePartner.svc
  5. After setting the key sharing location, the next step is to import the Trusted Publishing Domain (TPD). Do so by entering the following:
    Import-RMSTrustedPublishingDomain -RMSOnline -name “RMS Online”
  6. The final step is to activate the internal IRM licensing. Do so by entering the following:
    Set-IRMConfiguration -InternalLicensingEnabled $True
  7. After enabling IRM licensing, verify the functionality by entering:
    Test-IRMConfiguration -RMSOnline

The test should pass with the result OVERALL RESULT: PASS.

NOTE: Enabling IRM licensing might need several hours for the change to take effect. If possible, allow 8 hours to pass before proceeding with STEP 2.

STEP 2 – CREATE THE OME TRANSPORT RULE

We will create a transport rule that enables Office 365 Message Encryption if the message is sent to a recipient outside the organization and the Sensitivity header have been set to Confidential.

Follow these steps:

  1. Log in to your Office 365 Admin Portal and navigate to Exchange Control Panel (Admin\Exchange).
  2. Navigate to Mail Flow, click the + icon and select Create a new rule…
    Newrule
  3. Give the rule a suiting name and click More options…
    moreoptions
  4. From here you can set your condition as it fits your needs, but for this example we will inspect the Sensitivity header and apply Message Encryption based on that. To do so, select the following conditions:
    Apply this rule if… A message header includes any of these words
    amessageheader
  5. Complete the Apply this rule if-condition by clicking the properties Enter text and Enter word so that the condition makes ‘Sensitivity’ header includes ‘Confidential’
    sensincl
  6. Click Add condition and select The recipient… Is external/internal. Click Select one… and select Outside the organization and hit OK
    outside
  7. Proceed with clicking Do the following… and select Modify the message security… and select Apply Office 365 Message Encryption
    applyo365m
  8. Hit Save at the bottom of the New rule editor. (If you get the message that IRM licensing is not enabled and have completed STEP 1 – please allow more time for the change to take effect, as stated in the Note after STEP 1).

STEP 3 – SEND A CONFIDENTIAL MESSAGE WITH OME

We will create a message with the sensitivity level set to Confidential and send this to a recipient outside our organization. Our transport rule will apply Office 365 Message Encryption to the message.

Follow these steps:

  1. Open Outlook or Outlook Web App. Both clients have the native functionality to set the sensitivity header. In the example, I will use OWA, but the procedure is the same in Outlook.
  2. Compose a new message. Enter an external recipients, give the message a subject and some content, then click  and Show message options…
    messageopt
  3. Set the message Sensitivity level to Confidential and hit OK
    confidential
    NOTE: In the rich Outlook client, Sensitivity options are found under Message Options\More options\Sensitivity.
  4. Send the message.

STEP 4 – RECEIVE AND OPEN THE ENCRYPTED MESSAGE

  1. Open the inbox of the external mailbox and find the encrypted message
    protectedmessage
  2. As you can see, the message includes an attached HTML-document. Open the attachment.
  3. To be able to read the encrypted message, you can choose to sign in with a Microsoft Account or Corporate Credentials (AAD/Office 365 account) that has a username that matches the e-mail address to which the encrypted message was sent – or (new October 2014) you can choose to open the message with a One Time Passcode that is sent to the same email address that received the encrypted message (html file). I recommend using Organizational Accounts when sending to other Office 365 users, as they can then open the message with their corporate credentials or even an existing session cookie (Single Sign-on experience) – but for external users, which we are demonstrating in this case, One Time Passcode would be prefered. I am selecting the OTP choise by following the link at the bottom (“Don’t want to sign in? Get a one-time passcode…”).
    Select "Get a one-time passcode..."
  4. Now, go back to your inbox (recipient) to find the One Time Passcode and a reference code (handy if you have received multiple encrypted messages at once to differentiate the OTPs for different messages). Make a note of the OTP and go back to the browser.
    Find the OTP in your inbox
  5. Enter the Passcode from the email in the Passcode field, then click Continue.
    step2
  6. The message will now load and be displayed in the browser.
    voila

CONCLUSION

The Office 365 Message Encryption is a very good feature that allows Office 365/Exchange Online customers to send encrypted messages both inside and outside their organization. The new OTP functionality makes it even better and more complete. This post demonstrated the basic capabilities of OME, and you can make even more advanced Mail Rules, such as applying RMS templates to the message if it is sent to an internal recipient, and use Office 365 Message Encryption only to external recipients – just configure the rule to fit your needs.

You can also customize/brand the message that includes the HTML-file by using the
Set-OMEConfiguration cmdlets. The cmdlets allow you to set custom e-mail text, disclaimer, portal text and even include your company logo for branding.