Category Archives: Exchange Online Hybrid

Guidelines for Office 365 User Provisioning and De-provisioning Processes – The Lost Documentation

BACKGROUND AND PURPOSE

Introducing Office 365 and Azure AD in an Enterprise environment always raises the question around how object provisioning-, de-provisioning- and organizational change-processes will be affected. These are three common concerns and questions that I get from Enterprise customers in the planning phase of an Office 365 onboarding project:

  1. The customer is using an identity management (IDM/IAM) solution for provisioning accounts, and want’s to better understand how Office 365 / Azure AD will affect that landscape.
  2. The customer have special de-provisioning routines for retaining user data (i.e. keeping the mailbox and home folder data etc.), and needs to know how these processes should be updated when Office 365 is introduced.
  3. The customer understands the provisioning and de-provisioning processes, but want’s to automate them as much as possible.

The purpose of this article is to provide guidelines on how to approach this topic. Please note that these guidelines are intended for large Enterprises that seeks a high level of automation in these processes. Small-/medium-sized organizations with a lower number of employee attrition and/or organizational changes per week/month does not always have to implement automation processes for provisioning/de-provisioning/org changes.

As always with the “Lost Documentation” articles, most of this information is available from Microsoft, scattered over various TechNet- and MSDN-articles – this is my attempt to assemble the information in one place, to assist fellow Solution Architects, system administrators and project managers in the journey to Office 365.

PROVISIONING A USER FOR OFFICE 365 – WHAT COULD POSSIBLY GO WRONG?

Implementing user provisioning processes in Office 365 is generally very easy and straight-forward. Many often think that if you have complex provisioning processes in place today, you will have a hard time make it work for Azure AD / Office 365. This is almost never the truth, as the provisioning processes for Azure AD / Office 365 is “put on top” of what you already have. You kan keep provisioning users based on data from your HR system, and assign them roles based on data in a payroll system, or whatever it is that you are doing – as long as the user is created in Active Directory – it will be provisioned in Azure AD / Office 365 through a separate appliance-like synchronization engine (AAD Connect) without you having to change anything in the current provisioning routines. However, some processes will require updates – i.e mailbox provisioning in Exchange – and you will also be required to assign an Office 365 license to your users before they can use any of the services. The higher ambition you have for automation around these processes, the more challenging this topic will be for you – simply because automation is handled differently by every customer so there is no general answer to how it is achieved.

Below is guidelines for approaching provisioning automation, which can help you update your own routines and processes.

GUIDELINE #1: KNOW YOUR USER PROFILES BETTER THAN EVER BEFORE

The nature of SaaS: Office 365 is a subscription-based service, sold and consumed in a SaaS-model. Each user has assigned services in Microsoft’s data-center, with associated SLAs. A fundamental aspect of this delivery model is that each user must have an assigned license to be able to consume the service. A challenge for large Enterprise customers is that they are used to the traditional ways of purchasing Microsoft licenses (not services), which required a yearly inventory (“true-up”) of how many users were having Office applications, Exchange mailboxes, Skype for Business/Lync profiles, access to SharePoint etc. – and all you had to do as a customer was to share this updated figure with Microsoft – pay accordingly – and continue to use the services.

There have never before been any “enforcement method” that makes Office, Exchange, Skype for Business or SharePoint stop working if you are using more licenses that you have accounted for in your “true-up” – everything has just worked and it has been very much up to your own customer loyalty to make sure that you actually have payed for every user that you have provided with Microsoft software – such as Exchange mailboxes, Office applications, Skype for Business/Lync profiles and access to SharePoint Sites. And of course, this would include all your employees, but also any external user, like consultants getting Exchange mailboxes so that they could send and receive e-mail with your domain.

Don’t under-estimate the task: With Office 365 in place, the reality is changed. Users that are not assigned a license does not get access to any services (can’t activate the Office applications, no mailbox access in Exchange Online, no OneDrive for Business, no access to SharePoint Sites etc.). There is no “grace period” – you will need to have the right number of licenses reserved from your Enterprise Agreement through the VLSC Portal, and these licenses need to be assigned to a user (activated on that user’s account – read more later in this article) in order for that user to be able to use the services. This fact will require you to both know exactly how many users you are providing with Office 365 services, and if you are providing more than one license plan (SKU), i.e you have purchased a mix of E1, E3, E5 and K plans – you will need to know exactly what individuals should be mapped to exactly what license plan.

The task of mapping each user to a user profile (license SKU) is often extremely under-estimated. It is important to understand the consequences of not having the user profile mapping completed before commencing the Office 365 onboarding project. The consequence includes:

  1. The onboarding project will be stalled, as licenses will either run out, or the technical team doing the onboarding will at a certain point in time need to know what licenses to assign for each user, as all users are onboarded.
  2. The business case for moving to Office 365 will be impacted. Let’s say you invested in 50 000 E3 licenses, because that was the amount of Office ProPlus licenses you bought in the traditional license model. As it then turns out, you were actually providing these services for 55 000 users, you just had not calculated for all the external people who you have provided with IT-tools. This would mean an additional cost of 10% over your initial Office 365 investment.
  3. Users would get insufficient or wrong services assigned. Imagine if you assign E1 licenses to a department of 5000 users where you deploy Office 365 ProPlus – no one would be able to activate their Office applications, and your support organization would have to take the hit. Another consequence of getting the wrong services assigned is that you pay an extreme over-price for a user getting more services than he or she ever would actually use.

So before you dig deeper into the technology around provisioning, it is recommended that you map each and every user account in your organization to a user profile, which also translates to a license SKU.

Clean-up: Also note that accounts (i.e. Active Directory users with or without Exchange mailboxes) that are not actively used cannot be provisioned without an active license, which means that if you have licenses for 50 000 users, but you have 60 000 mailboxes in Exchange, where 5000 are Shared Mailboxes and Equipment/Room Mailboxes (these are free in Office 365 / Exchange Online), you have an overhead of 5000 user mailboxes which will need to either be removed or to acquire a license. Cleaning up your Active Directory and Exchange Server from inactive user objects and mailboxes before Office 365 onboarding is as important as the user profiling work itself.

GUIDELINE #2: UNDERSTAND IDENTITY/USER PROVISIONING AND LICENSING

Identity Bridge: The recommended way for Enterprise customers to provision users, groups and contact objects in Azure AD / Office 365 is by extending your on-premises Active Directory to the cloud through an appliance-like synchronization tool called Azure AD Connect (AAD Connect). The tool was previously known as DirSync and AADSync, and there was also a standalone FIM / MIM Management Agent – all these three still works, but future development will only be done in AAD Connect, which makes that the best choice for every organization today, regardless of complexity and identity landscape in general. Hand-in-hand with identity synchronization for provisioning and administration, a security token service (like ADFS) is usually implemented to provide single sign-on to Office 365 – this however has actually nothing to do with provisioning, why it will not be mentioned more in the article.

Sync: With AAD Connect in place, the on-premises Active Directory will be replicated to the Azure AD / Office 365 every 30 minutes. All your users, groups and contact objects will be provisioned in Azure AD / Office 365, but no Office 365 services are assigned to any users until you assign a licenses yourself as an admin. AAD Connect makes sure that you keep administrating your identities the same way you are today, and that you won’t have to maintain multiple identities (on the ground and in the cloud). It also makes sure that what ever identity provisioning routines you have in place today will continue to work as before, as long as they provision identities in Active Directory, AAD Connect will do the rest on top of what you already have.

Beware of 3rd-party or self-made sync: It is not recommended to replace AAD Connect with a 3rd party or self-developed synchronization / provisioning script (i.e. PowerShell / REST API provisioning). The main reason is that it will not be supported for all complex scenarios (such as Exchange Hybrid Configuration, attribute write-back scenarios, etc.). There is a AAD Federation Compatibility List (previously known as the “Works With Office 365 Identity Program”) that lists 3rd party STS services compatibility towards federation with Azure AD / Office 365 – found here – however, not many are reading the top note on the page that says that the solutions listed are only tested for federation scenarios – synchronization and multi-factor authentication scenarios provided by 3rd party vendors are not supported by Microsoft.

Synced users needs licenses: When AAD Connect is in place and is replicating your Active Directory data to Azure AD / Office 365, you will need to license users in order to activate them for Office 365 service – such as being able to active Office 365 ProPlus or read content from SharePoint Online. Assigning licenses can be done manually via the Office 365 Admin Portal, but if you want to automate this for Office 365 licenses, you have to create your own solution as of now. The recommendation is to implement a PowerShell-script that assigns licenses based on Active Directory attributes or Security Group membership. This script can run as a Scheduled Task, for example on the same server that hosts AAD Connect. A good place to look if you want to automate user licensing with PowerShell is at this post by Office 365 MVP Johan Dahlbom (and in this post he explains how to do it with Azure Automation).

Two things before you sync: It is recommended that you inspect your Active Directory for directory quality errors before you automate your identity provisioning and administration with AAD Connect. Use the IdFix tool to run the inspection and then remediate all errors found. It is also recommended that you change your users UPN-names to match their primary SMTP-addresses before you sync. You don’t have to change the samAccountName, so users can keep logging in to Windows etc. the same way as before. The main reason to why you should change UPN-names to match SMTP-addresses is that users should know what username to use when accessing Office 365 in the mobile context (it has to be in the UPN-format). If you have contemplated Alternate Login ID as the solution for this, beware of the limitations (great article here by Joe Palarchio) – I generally do not recommend using it.

GUIDELINE #3: UNDERSTAND MAILBOX PROVISIONING – DURING AND AFTER COEXISTENCE

With our user profiling, AD/Exchange clean-up, identity bridge and automated licensing in place, we are ready to look at process updates required for provisioning Exchange mailboxes.

Different ways depending on if you do hybrid or not: If you have established an Exchange Hybrid Configuration, you will provision mailboxes as an administrative task much like you have done before, until the last Exchange Server and the hybrid configuration is fully decommissioned. If you are not in a hybrid configuration, you will provision mailboxes simply by assigning a license for Exchange Online to one of your users. Let’s break down the rationales and guidelines:

As long as you are in an Exchange Hybrid Configuration with at least one Exchange Server still running on-premises, you can provision new mailboxes directly in Exchange Online by creating new “Remote Mailboxes”. A Remote Mailbox is a mailbox in Exchange Online that you can see and manage from your Exchange Server administration tools. You can of course create Remote Mailboxes from your Exchange administration tools (GUI), but for automation purposes it is recommended to use PowerShell. For updating your mailbox provisioning routines to create remote mailboxes in Exchange Online while you still have Exchange Server on-premises, study and use the PowerShell cmdlets for provisioning new Remote Mailboxes. Here are some examples:

To create a new Remote Mailbox for an Active Directory Mail User that you have provisioned on-premises, synced to Azure AD and assigned an Office 365 license (the most common scenario), run the following command in the Exchange Management Shell:

Enable-RemoteMailbox -Identity <UPN-name> -RemoteRoutingAddress <USERALIAS>@<TENANTNAME>.mail.onmicrosoft.com

To create a new user in the on-premises AD and a new Remote Mailbox that will be enabled when the new user is synced by AAD Connect (a fairly common scenario), run the following command in the Exchange Management Shell, and then license the user:

New-RemoteMailbox -UserPrincipalName user@domain.com -Name “Display Name” -OnPremisesOrganizationalUnit OU=Users,DC=addsdomain,DC=com -Password (ConvertTo-SecureString -AsPlainText “Pa$$word” -Force)

To create a new Shared Mailbox directly in Azure AD / Exchange Online as a Cloud ID (without any corresponding AD-user on-premises), run the following command in PowerShell, after connecting to Exchange Online:

New-Mailbox -Shared -Name <UPN-name> -PrimarySmtpAddress <User>@<DOMAIN>.com

(If you want the Shared-/Room-/Equipment Mailboxes to be represented in the on-premises AD, you will need to provision them as User Mailboxes first and convert them to shared mailboxes afterwards and remove the licenses – Shared-/Room-/Equipment mailboxes does not require a license once they are provisioned as such).

If you are not in an Exchange Hybrid Configuration, you provision mailboxes simply by assigning users with the correct attributes and an Office 365 license (that includes Exchange Online). The attributes that must be populated before you sync and license the users are: proxyAddresses (should include all necessary SMTP- and SIP-addresses), mail (should be populated with the primary SMTP-address), UPN (should preferably match the primary SMTP-address), Display Name (as you want it to appear in the Global Address List). The same process goes for Distribution Groups – you provision them in the on-premises AD with these attributes (minus UPN) populated, and then let AAD Connect synchronize them to Azure AD / Exchange Online.

About administration of Exchange Online: When AD synchronization with AAD Connect is enabled, most attributes are owned by AD on-premises, and you will therefore not be able to fully administer Exchange Online from the web-based administration portal (Exchange Control Panel). Some Exchange-attributes (such as proxyAddresses) simply must be administered in the on-premises AD and be synchronized to Azure AD / Exchange Online for the change to have effect. Because of this, it is worth noting that if you are decommission all Exchange Servers on-premises, you might need to build in more Exchange administration to your IDM / IAM provisioning and self-service tools than what you had before. End-users that administer Distribution Groups in Outlook are also affected, as this will for the same reasons stop working after the users mailbox is moved to Exchange Online – self-service tools for Distribution Group/List management will be required.

GUIDELINE #4: UNDERSTAND SKYPE FOR BUSINESS (LYNC) ONLINE PROVISIONING

Skype for Business Online (formerly known as Lync Online) provisioning works in the following way:

  1. If you have Lync Server or Skype for Business Server on-premises, and have users provisioned there with active SIP-profiles, and you are using AAD Connect so sync your AD on-premises to Azure AD / Office 365, no provisioning will occur in Office 365 for the SIP-enabled users, regardless if you license these users for Skype for Business Online in Office 365.
  2. If you license SIP-enabled users for Skype for Business Online, and then remove the SIP-profile from Lync-/Skype for Business Server on-premises, users will be provisioned for Skype for Business Online at the next AAD Connect sync cycle (often referred to as a “cut-over move”).
  3. If you are leveraging a Lync-/Skype for Business hybrid configuration, users must be licensed for Skype for Business Online, and you can then move users from the on-premises pool to the online pool. When all users are moved, you can for the purpose of provisioning decide to keep or remove the on-premises Lync-/Skype for Business Server environment, and select another provisioning method from this list of guidelines.
  4. If you are not having Lync-/Skype for Business Server on-premises, you provision users for Skype for Business Online simply by assigning them an Office 365 license (with Skype for Business Online included). It is recommended that you in this case populate the proxyAddresses attribute in the on-premises AD with a SIP:-address that matches the primary SMTP address.

GUIDELINE #5: UPDATE YOUR PROCESSES FOR DE-PROVISIONING USERS

There are many different aspects of user de-provisioning, including return of licenses, removing permissions, store/hold user data, etc. By understanding the fundamentals of how Azure AD / Office 365 de-provisioning works, you will be able to update your current processes to work with Office 365. Here are the fundamentals you should know about:

  1. If you are disabling a user in the on-premises AD, you are disabling the user in Azure AD / Office 365, but the license is still active.
  2. If you are deleting a user from the on-premises AD, you are deleting the user in Azure AD / Office 365. When this happens, all data is soft-deleted for 30 days and the license is returned to the license pool. If you restore the user within these 30 days (restore the user in the on-premises AD and sync the user again), all data will be kept. If you don’t restore the user within 30 days, and have not applied any rules for data Hold, the data is lost.
  3. If you remove the license from a user in Office 365 (for example if you move the user to a OU for de-provisioned users, disable the account and strip it from it’s group memberships as your de-commissioning routine in the on-premises AD), all data is soft-deleted for 30 days. If you re-assign the license to the user within the 30 days, and have not applied any rules for data Hold, all data will be kept. If you don’t re-assign the license within 30 days, the data is lost.
  4. If the user is licensed when it is deleted and the Manager field is populated for that user in AD, the Manager will be notified and given access to the users OneDrive for Business site for 30 days through the Access Delegation mechanism, explained here. If you remove the license before actually deleting the user, the Access Delegation process will not be triggered.
  5. There are different ways to keep the data for more than 30 days after you have removed the license from a user. The general recommendation is to use the Hold capabilities, administered from the Office 365 Protection Center.

GUIDELINE #6: UPDATE YOUR PROCESSES FOR ORGANIZATIONAL CHANGES

When users go through a department change, country transfer or other forms of organizational change, there are often identity-related processes that goes with it. In Azure AD / Office 365, it is much like with the provisioning; you can keep doing what you did before and let AAD Connect handle all the provisioning aspects in Azure AD / Office 365 for you. There is however one concern regarding the UPN attribute that you should be aware of. Here are the fundamental guidelines for organizational changes:

  1. If the organizational change is resulting in attribute updates, or even changes of Organizational Units, in the on-premises AD – let AD Connect take care of updating this in Azure AD / Office 365 for you through the regular synchronization cycle.
  2. Beware of delays that can occur, such as generation of the Offline Address book used in Outlook that can make updates in the Global Address List need a day to show up to all users, or back-end replication to SharePoint Online to let all organizational attributes show the correct values.
  3. If the organizational change requires a change of the UPN-name and the user is licensed, you will need to manually give it a push in Azure AD in order for it to change, AAD Connect can not change UPN-names in Azure AD / Office 365 for licensed users. To manually fix the UPN-name for the licensed user, follow the steps outlined in Scenario 2 in this article. A summary of the steps: change the UPN-name in the on-premises AD, let AAD Connect run a sync cycle, then connect PowerShell to your Azure AD tenant and run this command in PowerShell:

Set-MsolUserPrincipalName -UserPrincipalName [CurrentUPN] -NewUserPrincipalName [NewUPN]

SUMMARY AND DISCLAIMER

All guidelines presented here are based on my field experience and are made available for your own interpretation to assist in updating provisioning/de-provisioning routines in the move to Office 365. I hope you will benefit from these guidelines, and that you will give me suggestions for more topics that would make this article more complete. I am always listening to feedback on Twitter. Thank you for reading.

Managing Change in Enterprise Office 365 Implementations (or “Do We Need a Test Environment?”) – the Lost Documentation

BACKGROUND AND PURPOSE

One of the top challenges for Enterprise customers in their journey to Office 365 and Cloud based services is the transformation needed to handle the constant service Changes that make out a natural part of SaaS and the Cloud. The concept of “evergreen services” is still for many completely new, and customers often finds this parameter more challenging than the technical implementation and onboarding itself. Microsoft recognizes this, and are committed to help organizations prepare for change by implementing various change control mechanisms to the service. They are also more transparent to their customers than they have ever been in regards to Product roadmaps (see the Office 365 Public Roadmap here) and Community presence (see the Office 365 Network on Yammer here). Regardless of all these efforts from Microsoft, Enterprise customers that are new to Office 365 always asks me and my team for advice in this complex topic. The question is initially almost always formulated in the same way: “How do we create the test Environment for this?”. The answer: “It’s not that simple…”, and then the discussion follows, uncovering all aspects from handling operational maintenance to planning for unavoidable updates in the user interface.

The purpose of  this post is to give an overview to how Enterprise customers should comprehend and plan for ongoing change after Office 365 is implemented and all the users are onboarded to the service. Microsoft provides good information in their service descriptions, on the Office Blogs etc, but this post is to be considered as “the lost documentation” of the holistic overview that often is requested by customers when they are new to Office 365.

OFFICE 365 IS “EVERGREEN” – WHAT COULD POSSIBLY GO WRONG?

The concept of Evergreen means that the service will constantly improve and evolve, without you as a customer having to worry about it. The reality of it is that you do not need to take care of server version upgrades, security patches, feature implementation or maintenance – your service provider takes care of that for you. This will save you time and money, not least as you never have to run an upgrade project again for i.e Exchange Server or Skype for Business Server etc. On the other hand, if your organization is not used to rapid releases and always being on the latest version, you will need to shift your mindset from “big change projects every five years” to “smaller updates every month”. The concept is often referred to as moving from “waves to ripples”, a good metaphor to how things will change after the move to Office 365.

Evergreen is great, however it will put new requirements on your IT-organization if you are not used to deliver changes rapidly. This scares many Enterprise customers, and sadly some are disqualifying Office 365 just because they think that they can’t control change in Office 365, and that “a high release pace determined by somebody else” is therefore nothing for them. That is a false apprehension, as there are many ways to control change in Office 365, which will be covered in this post.

As the World is changing and millennials are entering the workplace, the pressure from the business on the IT-organization to deliver a modern digital workplace is often very high. Also, providing modern productivity- and collaboration tools is often key to prevent rogue IT, where users are adopting Commercial solutions for cloud storage, or send sensitive information to Commercial e-mail services so that they can access it anywhere on any device etc. These business requirements goes hand in hand with what Office 365 offers, so the IT-organization will need to deliver, and at the same time manage change in an Evergreen service. Let’s have a look how this can be accomplished in practice.

BECOME AN “EVERGREEN READY ENTERPRISE” WITH OFFICE 365 – CATEGORIES OF CHANGE

I will explain how to manage change in Office 365 by breaking the topic down to categories. I will explain the challenges and remediation options for each category. If you have ideas for other categories, please reach to me on Twitter (@JesperStahle), and I will consider adding them to this post. My intention is to keep the documentation alive and updated as Office 365 is changing.

CATEGORY #1: THE ROLE AND RESPONSIBILITY OF THE SERVICE OWNER

CHALLENGE: The shift to become “Evergreen ready” starts with the role of the Service Owner (sometimes referred to as Service Responsible or Service Manager, depending on how ITIL-influensed operations model you might have). The person owning the Office 365 service, regardless if it is the whole suite, or independent services (such as Exchange Online, SharePoint Online, etc.), must be prepared to become proactive in a whole new way than before. Traditionally, the Service Owners might find themselves in a situation where there are several months or even years between service changes, and the role is often described as more reactive than proactive. When a change is needed due to new dependencies or business requirements, the Service Owner would be notified and take the necessary decisions and issue transformation projects.

When the Service Owner becomes responsible to deliver an Evergreen service within agreed SLAs, it will be required that he or she becomes more pro-active to be notified of upcoming changes, before they are implemented and affects end users and the IT-operation. If the notifications are missed or ignored, the Service Owner will one day have a change issued outside of their control, which might cause frustration from both IT-operations and end users, and in the worst case even impact the SLA for which the Service Owner is responsible for.

BECOMING PROACTIVE ENOUGH: It is vital that the Service Owner of Office 365 suite, and/or its components, becomes proactive by monitoring all the communications from Microsoft regarding upcoming changes. Microsoft will not implement changes in Office 365 without first notifying their customers, but if you do not have a proper ownership and process for reading and responding to these communications they are to no use for your organization. It is a bad practice to trust the administrators of Office 365 to monitor the communications about upcoming changes and react to them accordingly – the administrators should put their focus on administering the service and enabling new or changed features as that is ordered/issued by the Service Manager. A better practice is to have a named Service Owner that takes responsibility for service communications from Microsoft as well as monitoring the Office 365 Roadmap to plan for upcoming changes and feature introductions.

In practice, this means that the Service Owner should monitor four main communication channels and implement processes to be able to rapidly respond to the changes announced in them. Here is the list of the four main communication channels:

  1. The Office 365 Message Center: In the Office 365 administration portal there is a Message Center that is used to notify customers about upcoming change and incidents. The Service Manager can access the Message Center by acquiring an administrative role in the Office 365 tenant (i.e the Service Admin role) then visit the Office 365 Admin Portal and navigate to Message Center. Here is a direct link. Here is an example screenshot of how the Message Center looks like (click picture to enlarge):The Office 365 Message CenterAs you can see, there are different categories of information. Not every communication in the message center is necessarily targeted for the Service Owner or similar function. Here is an overview of the Message Center taxonomy (click picture to enlarge):The Message Center TaxonomyIt is up to every organization to shape their own RACI matrix for these communications, but it is generally recommended for the Service Owner to be responsible for communications classified as “Stay informed” and “Plan for change”, and to be informed about communications classified as “Prevent/Fix issue” and “Service Incidents”.
  2. The Office 365 Public Roadmap: The next communications channel the Service Owner should be responsible to monitor is the Office 365 Roadmap, found here. This is a great evidence that Microsoft is aware that change requires planning. The roadmap site let customers see which features are in development and which that are currently being rolled out. When features are due to be rolled out, you will also be notified in the Message Center, so the main reason why the Service Owner should have the ownership and responsibility to act on this communication channel is for the organization to be aware about features that are in the “In development” category. By monitoring this category, the Service Owner can avoid surprises in the Message Center, and will have much more time to plan for change. Here is an example screenshot of how the Office 365 Public Roadmap looks like, note the category “In development” (click picture to enlarge).Office 365 Public Roadmap http://roadmap.office.com
  3. The Office Blogs: The third communication channel every Office 365 Service Owner should monitor is the Office Blogs – found here. Here you will find various communications from the teams at Microsoft that build Office 365, not just feature announcements, but also interesting costumer stories, news round-up articles and video casts (Office Mechanics) covering different a wide variety of topics. A good advice is for Office 365 Service Owners to have the Office Blogs as their start page in their web browser.
  4. The Office 365 Network on Yammer: With over 89 000 members and growing, the Office 365 Network is a community for Office 365 customers and IT-professionals world wide. Service Owners can participate in various groups within the network to discuss Office 365 topics. Microsoft themselves has a strong presence here, and often customers find their questions answered directly by representatives from the Microsoft product groups. This is a great forum for Service Owners as they can learn how others are solving issues that they are facing. The recommendation for the Service Owner is to monitor the network (and/or it’s quarterly digests known as the Office 365 Network Bulletins), but it is not necessary to read all its content daily. The Office 365 Network is found here.

CATEGORY #2: PROCESSES FOR SERVICE UPDATES, FEATURE INTRODUCTIONS AND UI CHANGES

Release Options: In Office 365, updates and new features are released constantly. Staying informed through the communication channels helps you prepare for change, but in order to be fully prepared processes are required for testing and evaluating new functionality. Therefore, it is possible to nominate users in the organization to be part of the First release ring (wave) for updates. First release users will have new features and updates before other users, so that they can test the updates in production and prepare integrations and documentation, such as end-user communications.

It is also possible to nominate all users in the tenant to be part of the first release ring by enabling First release for scope of the Entire organization. That would give everyone access to new features sooner, but will give the Service Owner and the IT-organization less time to prepare for change, which is not optimal for Enterprise customers. In May 2015, around 10% of all Office 365 customers had enabled First Release for the Entire organization, not just for selected people.

Find more information about the First Release options here. Here is an example screenshot of the Office 365 release options and how they are configured in the Office 365 admin portal (click picture to enlarge).

Release Options in Office 365

End-user awareness: Upcoming change is announced through the communication channels, and will be available for test and evaluation for the users nominated to be part of the First release ring. Those individuals should in concert with the Service Owner evaluate all dependencies to each new feature and update, as well as assess if end-user communications will be necessary. Where end-user communication is necessary, Microsoft often provide templates and input for your organization, so you only need to review the material and make your finishing touches to it.

The Skype for Business example: A good example of a change process was when Microsoft changed the name and user interface to Skype for Business from what was formerly known as Lync. Microsoft first announced this in the Roadmap site and on the Office Blogs, then shortly after in the Message Center under the category “Prepare for Change”. All the information needed was provided for the Service Owner to plan for change, and a preview version of Skype for Business was made available to install for the First Release users (or anyone else who wanted to try it). Microsoft prepared end-user information in both written form and in video format (see picture below) for customers, ready to be distributed directly to end users. The article with that material can be found here for reference. If any customer had any questions about the update – before, during or after – they could ask them in the Office 365 Network on Yammer and have the discussion going, and a potential answer, moments later.

Example of end-user material provided by Microsoft when Lync became Skype for Business

CATEGORY #3: PROCESSES FOR OFFICE 365 PROPLUS UPDATES ON CLIENTS

Whatever you did before will work: A common misunderstanding is that if you deploy Office 365 ProPlus (Click-to-Run), you have no control of updates and your users will have features introduced without your control. The reality of it is that any change/update process you had for MSI-based distribution of Office can be used with Office 365 ProPlus Click-to-Run, you just have to adjust your distribution method a bit. You can distribute Office 365 ProPlus using SCCM (good information in this article) and you can then distribute updates in a staged fashion with test rings/waves, just as you probably did before. If you have SCCM Version 1606 or later, you can distribute the updates using SCCM (see details below).

Control distribution of updates without SCCM: When deploying Office 365 ProPlus, you can use a configuration file (config.xml), or Group Policy settings, to configure the location for updates for individual computers using the UpdatePath parameter. Read more about the config.xml parameters here, and the GPO settings here. The UpdatePath will point to a file share on the network where binaries for the latest available build of Office 365 ProPlus will be located. It is up to every organization to plan and design the file share – you can use shared folders on a DFS file share or individual file shares distributed on site-local file servers etc. You can have separate file shares for pilot groups that can evaluate the Office updates (you place new updates here first), and these updates can then be moved to the production file share that are used by the majority of the users. This creates an environment where you can stage the updates and evaluate them before distributing to the masses, which also facilitates end-user communications if needed for user interface changes etc. To generate the config.xml, I recommend using this XML Editor tool.

Control distribution of updates with SCCM: If you have SCCM, you can use it to distribute updates. It is recommended to use the methods explained here on SCCM 1606 or later, but if you are on version 1511, you can use functionality that will remain in preview explained here.

Updates can be postponed, but to stay fully supported you should deploy Office 365 ProPlus updates within 12 months from when they were released. Security updates and feature updates are today bundled together, but options for separating them will be available soon. Regardless of that, the recommendation is to evaluate updated versions in the pilot ring immediately when they are available, and distribute them to the production ring as soon as possible after the verification. Consuming cloud services requires the client to be up to date to have full support, optimal performance and all features available – try not to fall into old habits and postpone updates just because you have the option to do so for 12 months.

First Release users (see Category #2) will have the option to download Preview versions of Office 365 ProPlus to test functionality that is due to be released in a bit more distant future. The First Release users can sign in to the Office 365 Portal and download the Preview version from the Software section (direct link here) – just scroll down on the page, the Preview version is located under the download section for the current version. The Preview versions will never be displayed for users that are not in the First Release ring.

An example overview of the non-SCCM solution (Group Policy based): In the graphics you can see a file server hosting two file shares containing different versions of Office 365 ProPlus – one for the Pilot Ring and one for the Production Ring. The binaries (bits) are downloaded from Office 365 on the server (can be automated with scheduled task). Users in the Pilot Ring have their UpdatePath parameter set to the Pilot-share, and the users in the Production Ring have their UpdatePath parameter set to the Prod-share. In the First Release Ring, users are downloading Preview bits from the Office 365 Portal under the Software section. Click graphics to enlarge. Note: If you have SCCM, the native methods referred to above (“Control distribution of updates with SCCM”) is recommended.

 

 

Overview of how Office 365 ProPlus updates can be distributed in the Enterprise.

CATEGORY #4: MANAGING CHANGES TO YOUR ON-PREMISES FOOTPRINT (ADFS, AADSYNC, EXCHANGE HYBRID, ETC.)

Enterprise Office 365 deployments are integrated with the on-premises infrastructure through an “identity bridge” (synchronization and federation, also known as AADSync and ADFS) and sometimes an Exchange Hybrid configuration. There can also be integrations with Lync/Skype for Business (Hybrid Configuration) as well as Search integrations between SharePoint Server on-premises and SharePoint Online.

Test environments for this can be provisioned, but it does not makes sense to do so if the on-premises system in question does not have a full test environment to begin with. As an example, if you don’t have a test environment for Active Directory before introducing the identity bridge to Office 365/Azure AD, it does not make sense to provision a test environment for the AADSync and ADFS components, as they would read and write to the exact same source and target as the production instance. If, however, you do have a full featured test environment for Active Directory, you could of course choose to provision a separate Azure AD tenant to which you integrate the test environment through an identity bridge in the test environment.

The same logic goes for Exchange, Lync/Skype for Business and SharePoint – if you don’t have proper test environments to begin with, it would not be possible to produce a fair test environment for the Office 365 integration.

If the intention is to create a test environment just for AAD Connect and the Office 365 tenant, Microsoft have published step-by-step instructions on how to achieve that in a Microsoft Azure environment here.

Different support statements are true for the on-premises components. Here are the main guidelines to follow:

  1. AADSync / DirSync / AADConnect: The installation of the sync tool is meant to be “appliance like”. You do not necessarily need to update to every version that is released to stay supported unless you’re noticed to do so (all though it is generally recommended to stay up to date from the feature- and performance perspective). You can choose not to upgrade the sync tool until you are implementing a certain feature that requires a certain version – or until you are receive a message in the Message Center (see Category #1) saying that you need to upgrade to stay supported. The general recommendation is for the Active Directory Sync Tool Service Owner to monitor the Version Release History article, found here, to be aware about when updates are released and plan for installing the update accordingly. If test environments are not used, there are is a “safety net” measure that customers can take to prevent configuration errors in the sync tool to soft delete users in Azure AD. It is called PreventAccidentalDeletes, more information can be found here. As mentioned above, a test environment for AAD Connect and Office 365 tenant can be built in Microsoft Azure following the steps outlined here and here.
  2. ADFS: Like with Active Directory and Windows Server in general, it is recommended to keep ADFS updated through Windows Update. This is recommended for security reasons and to ensure optimal performance and functionality. So far, all versions of ADFS that have been released since Windows Server 2008 (ADFS 2.0) have been supported in Office 365, so you may consider your installation fully supported as long as you are installing all Windows Server updates from Windows Update, and as long as the version of Windows Server that you are running is in Mainstream Support (find out more here). However, new ADFS-features are available with new versions of Windows Server, which may encourage you to upgrade your ADFS environment as well. An example of such feature that have made many ADFS 2.0 (Windows Server 2008/2008R2) customers upgrade to ADFS 3.0 (Windows Server 2012R2) is the Extranet Lockout feature (read more here).
  3. Exchange Hybrid Configuration: To stay supported in an Exchange Hybrid Configuration, your on-premises Exchange Server environment must be on a supported version (at the time writing Exchange 2007 SP3RU10, 2010 SP3 and 2013 CU7, where 2010 and 2013 can be the Hybrid servers facing Exchange Online) and the latest available build minus one cumulative update should be installed – also referred to as the “N to N-1 guideline”). This means that if you have an Exchange Server 2010 SP3 environment on-premises, and the latest available version is Exchange Server 2010 SP3 Update Rollup 10, you would have to have at least Exchange Server 2010 SP3 Update Rollup 9 to be fully supported. The written support statement can be found here, and it actually does not mention the N to N-1 guideline. However, if you run in to issues and attempt to open a support ticket, Microsoft will remind you of the N to N-1 guideline and encourage you to update to the N to N-1 build and reproduce the issue on that version before troubleshooting continues.
  4. Lync, Skype for Business and SharePoint Hybrid Configurations: For these workloads, the support statements are not as clear as with Exchange etc. The general recommendation is to keep the on-premises server products on the latest available build (install available updates) and monitor the Message Center for notifications about supportability.

CATEGORY #5: DEVELOPMENT, CUSTOMIZATIONS AND BRANDING

There is one category where test tenants makes absolute sense – and that is for system development. In the perspective of Office 365 as a platform for developers, providing a test tenant is absolutely necessary. Everything from branding the suite bar to releasing code in SharePoint Online benefits from being tested first. However, these test tenants do not require a full identity bridge etc to do their job – instead developers can provision their own test tenants from their MSDN Subscription benefits (or even Trial plans), or you can provision one on a separate subscription. This test tenant can be completely out of scope for the Service Owners responsibilities. As mentioned for AAD Connect above, another option for creating a development environment is to leverage Microsoft Azure by following the steps outlined here.

SUMMARY AND DISCLAIMER

There are many misunderstanding when it comes to Office 365 and managing change in the Enterprise. Hopefully this article can help som organizations realizing that the Cloud, SaaS-apps and “Evergreen services” are manageable even in an Enterprise environment, and that good instruments exist so that the Service Owner can master even a very rapid release pace. Please note that the information given in this article is to be considered as guidelines shaped from real-world experience – but that change management is a complex topic that always have organization specific requirements. Modernizing change management to be “Evergreen ready” is not always easy, and it will most probably not happen over one night – but it is absolutely possible. Hopefully content of this article has helped proven that for you.

And remember: change is good!

Lync / Skype for Business Client Asks for Credentials (“EWS not Deployed”) in an Exchange Hybrid Configuration

BACKGROUND

Consider the following scenario:

  1. You have successfully established an Exchange Hybrid Configuration between your on-premises Exchange Server installation and Exchange Online.
  2. You are using a fully functional Lync / Skype for Business Server on-premises installation for all users.
  3. DNS zones for any of your primary SMTP domains exist in the internal network DNS (split-DNS is used)
  4. Some users are moved to Exchange Online, and some are still residing in the Exchange Server installation on-premise.

The following symptoms occur:

  1. For users that are migrated to Exchange Online, sporadic credential prompts appear from the Lync / Skype for Business (SfB) client, asking users for username and password. Users that still have their mailbox residing in Exchange Server on-premise are not asked for credentials.
  2. The following message is displayed in Lync / SfB Configuration Information (right-click the Lync / SfB icon in the System Tray to find this option): EWS Information: EWS not Deployed

CAUSE

Two ways to update presence information: The Lync / SfB client is by default configured to update your presence based on your Exchange calendar (Personal information manager settings in the Lync / SfB client). Lync / SfB can read your presence information from Exchange by either receiving it through the Exchange Web Services (EWS) API, or by reading it from Outlook (if Outlook is installed and running). The EWS method is dependant on Exchange Autodiscover to be fully functional (details will follow under Resolution).

If, as in the scenario explained above, the Configuration Information indicates “EWS not deployed”, the Lync / SfB client will fail to use the EWS API to update presence information. Instead it will fall back to updating the presence information from Outlook.

But it worked before: The Outlook-method have so far worked fine, and will continue to work fine, for non-migrated users as the Lync / SfB client is running under the same user context as Outlook. When users are migrated to Exchange Online and configure Outlook to run in the Office 365 context (regardless if using ADFS/SSO or not), the Lync / SfB client will no longer be able to update the presence information from Outlook without first having you authenticate in the Lync / SfB client with the Office 365 credentials. Migrated users are no longer running Outlook and Lync / SfB in the same user context. This produces the credential prompt.

To resolve this, the EWS method should be used instead. The “EWS not deployed” error must be repaired.

RESOLUTION

Exchange Autodiscover is key: The Lync / SfB client will use Exchange Autodiscover to resolve the URL for EWS. Exchange Autodiscover is assumed to be working in this scenario as the Exchange Hybrid Configuration is established (which itself have a Heavy dependency to Autodiscover). If Exchange Autodiscover is a new topic for your, I suggest reading this article before continuing.

SCP vs. DNS: Outlook will use Active Directory Service Connection Points (SCP) to find the URL to Autodiscover on the internal network (Active Directory joined machines). Outside the internal network, Autodiscover URLs are distributed using public DNS (looking at the domain in your e-mail address and ask for Autodiscover records in that domain). As internal clients rely on SCP records in Active Directory, there have never been any need to add the DNS records for Autodiscover in the internal DNS zone (the split-zone) for the SMTP/SIP domains.

The Lync / SfB client will not use Active Directory SCP records to retrieve the Autodiscover URL. It will only use the DNS-record method. This differs from Outlook, which can use SCP records to find Autodiscover. Because of this difference, you might have a fully functional Outlook (finds Autodiscover via SCP-records) and a fully functional Exchange Hybrid Configuration (finds Autodiscover via A/SRV/CNAME records in the public DNS zone), but you will not have a functional Lync / SfB client (attempts to find Autodiscover from the internal split-zone for the SMTP/SIP domains).

How to fix it: In order for the Lync / SfB client to be able to find Autodiscover on the internal network, add A/SRV/CNAME records (whatever mirrors your public DNS records) in the internal DNS zone (the split-zone) to mirror the setup for Autodiscover records in the public DNS.

After adding internal DNS records for Autodiscover, the Lync / SfB client will be able to find Autodiscover and thus the URL for EWS. Through EWS, the Lync / SfB client can retrieve your presence information from the Exchange Online mailbox via the Exchange Hybrid Configuration, and users migrated to Exchange Online will no longer suffer from credential prompts appearing in the Lync / SfB client.

Free/Busy not working from Office 365/Exchange Online to On-Premise Exchange / Hybrid Configuration – “The external recipient’s server could not be determined” “No information” (You forgot to iisreset)

BACKGROUND

You have successfully deployed a Hybrid Configuration between your on-premises Exchange 2010 SP3 Organization and Exchange Online. Everything works as it should, except for Free/busy look-ups made from Exchange Online mailboxes to a mailbox that still resides on-premise.

You have verified that the following is OK:

  1. You have run ExRCA diagnostics using https://www.testexchangeconnectivity.com/ , selected the Office 365 tab and then Free/busy tests. You simulate Free/busy lookup from Online to On-premise and it works with all green lights.
  2. You have also followed all the troubleshooting steps presented in this Microsoft Knowledge Base article: http://support.microsoft.com/kb/2555008 – all with successfull results.

Still, when attempting to do Free/busy lookups from Online to On-premise, you get the message in Outlook or OWA saying “No information” with the reason “The external recipient’s server could not be determined”.

CAUSE

Settings made during your Hybrid Configuration and/or extended troubleshooting of this error have not been fully commited to Internet Information Server (IIS). Allthough the diagnostics (ExRCA etc) say that your configuration is right, vital configuration is not yet effective.

RESOLUTION

First step in resolving Free/busy errors is to follow the two steps mentioned in the Introduction to this post (ExRCA diagnostics and following KB2555008 – see above). If that does not work:

Doing iisreset on your CAS server publishing EWS/Hybrid could fix this issue.

Plan for a very short outage in Exchange services (OWA/EWS etc) and open the command prompt on your Hybrid CAS server. Then run the command iisreset

After doing IISReset, Free/busy lookups from Online to On-prem should start working within minutes.

I have seen this in the field where we spent days troubleshooting this issue on Exchange 2010SP3 hybrid configurations after finally solving it with this basic, but classic, method – restarting the services. There could of course be other reasons behind the error (i.e related to how Autodiscover is published to the internet), iisreset should not be considered as a “general cure”, but a necessary troubleshooting step if your configuration is done by the book and it still does not work.

This mailbox exceeded the maximum number of large items that were specified for this request. (Fatal error TooManyLargeItemsPermanentException has occurred.)

POST UPDATED TO REFLECT THE EXCHANGE ONLINE ONBOARDING SIZE LIMIT INCREASE (FROM 35 to 150 MB) ANNOUNCED JANUARY 2015 (INFO).

BACKGROUND

When moving mailboxes to Exchange Online, the Remote Move Request fails with the following error message in Exchange Management Console (EMC):

Error: This mailbox exceeded the maximum number of large items that were specified for this request.

When viewing the log file for the mailbox migration, you see the following details:

Fatal error TooManyLargeItemsPermanentException has occurred.

CAUSE

The message size limits for Exchange Online migrations is 150 MB. It used to be 35 MB prior to January 2015, but have now been increased (info). If larger items than 150 MB are attempted to be moved through a default Remote Move Request initiated from the EMC wizard, the job will fail with the error message above because Exchange Online refuses the messages due to their size. If you have mailboxes containing messages that are +150 MB in size, you can still migrate these mailboxes but you will need to tell Exchange that it should simply skip the large messages that are refused and not abort the complete move request when encountering them.

RESOLUTION

If a mailbox has items that are 150 MB+ in size, you can solve this error in two ways:

  1. Either instruct the end-user to remove the item. Send an instruction on how to sort items on size in Outlook and tell them to Shift+Del Everything above 150 MB in size. Elite users can maybe even handle an instruction on Search Folders/Filter. After the users have removed the items, remove the Move request and start over again (resuming it will fail).
  2. Or, allow the move request to skip the 150 MB+ items that are refused by Exchange Online and continue the migration without aborting the entire job. Here is how:

Unfortunately, you cannot use the EMC wizard for this, you will need to use the Exchange Online Control Panel (where the migration wizard will allow you to adjust the parameters) to run the migration or use PowerShell. Using the Exchange Online Control Panel works for both Exchange 2010 and Exchange 2013 hybrid configurations.

2.1: USING THE EXCHANGE ONLINE CONTROL PANEL INSTEAD OF EMC (RECOMMENDED)

The most simple way to solve this problem is to run the migration from the Exchange Control Panel (as said, this works with both Exchange 2010 and 2013 hybrid configurations). Follow these steps:

  1. Log in to you Office 365 Admin Portal and navigate to Exchange Control Panel (Admin\Exchange).
  2. Navigate to Recipients (left pane) and then the tab Migration (top pane/tabs)
  3. Click + (plus) and Migrate to Exchange Online
  4. Select the first choice to initiate a remote mailbox move
  5. Select users to migrate or use a CSV to create a new migration batch
  6. At the next step, enter your MRS endpoint, that is where EWS is published (your hybrid server). Example: ews.company.com
  7. At the next screen, you can give the batch a name for tracking and logging purposes, select if you want to migrate an Archive Mailbox, and most importantly set the allowed number of large and corrupt/bad items to be skipped for the migration. Increase this number to be able to run the migration that failed. As an example, you can set both values to 50.
  8. At the next screen, enter who will have reports when the migration batch is completed, set scheduling options (start now or later) and select if the migration should be completed directly or if it should be automatically suspended when the mailbox replication have reached 95%, so that you can switch over the user when you want in a controlled manner. If using the auto suspend option, the user will not notice anything until you choose to complete the migration, then the last 5%/delta will be synchronized and the user will be switched over (email will be forwarded to Exchange Online mailbox and Outlook will re-configure for use with the Exchange Online mailbox etc).
  9. When completing the wizard, you can then track the migration from the Migration view in the control panel. If you are using automatic suspend, here is also where you can complete the migration after reaching 95% (mark the job and select complete).

2.2: USING POWERSHELL

As stated above, using Exchange Online Control Panel is recommended. If you would like to use PowerShell instead, follow this method.

If you are moving just one user, follow these steps:

New-MoveRequest -remote -Id firstname.lastname@smtpdomain.com -RemoteHostName “ews.companydomain.com” -RemoteCredential $Cred -TargetDeliveryDomain ‘tenantdomain.mail.onmicrosoft.com’ -SuspendWhenReadyToComplete -LargeItemLimit 50 -BadItemLimit 50

Note that I added LargeItemLimit and BadItemLimit. These parameters will allow up to 50 large (over 150 MB) messages to be refused and skipped, and up to 50 corrupt messages to be skipped as well.

If you are moving several users, you could use a CSV file with a column that says UPN, filled with your migration batch users UPN-names:

Import-Csv C:\MigrationBatch.csv | foreach {New-MoveRequest -Identity $_.UPN -Remote -RemoteHostName ews.companydomain.com -TargetDeliveryDomain ‘tenantdomain.mail.onmicrosoft.com’-BadItemLimit 50 -LargeItemLimit 50}

For both single and bulk, consider adding the following parameter to use the AutoSuspend feature:

 -SuspendWhenReadyToComplete -Suspend

Moving mailboxes with EMC is very convenient, but it will not give you the option to allow corrupted or large (150 MB+) items to be skipped and overseen in the migration job. Go Exchange Online Control Panel or PowerShell to save the day.