Category Archives: SSO

ADFS 3.0 and Hardware Load Balancing – the Lost Documentation

Article updated October 2014 to include http based health check probe as a workaround, made available with August 2014 Update Rollup for Windows Server 2012 R2.

BACKGROUND AND PURPOSE

ADFS and Federated Identities are common components for Enterprise Office 365 customers necessary to allow Single Sign-on, use of Claim Rules and immediate account lock-outs etc. As the availability the ADFS service decides the availability of Office 365 (if you can’t authenticate you can’t use the service), load balancing is a must-have. In most cases, Hardware Load Balancing (HLB) is used, and for ADFS 2.0 (Windows Server 2008/2008R2) and ADFS 2.1 (Windows Server 2012) this have never been a problem. You deployed your ADFS STS and/or ADFS Proxy, and published them with your Hardware Load Balancer like “any other SSL site” and it just worked.

Today, ADFS have with Windows Server 2012R2 reached version ADFS 3.0 – and the full “out-of-the-box-support” for using “any Hardware Load Balancer” is not as obvious any more. Making it  work is a bit more tricky, depending on what HLB you are using. The purpose of this article is to present my lessons learned from deploying ADFS 3.0 with HLB – and to save you from hours of trouble shooting when getting your own ADFS 3.0 HLB up and running with Office 365.

ADFS 3.0 AND HARDWARE LOAD BALANCING – WHAT COULD POSSIBLY GO WRONG?

ADFS 3.0 and Web Application Proxy (WAP) in Windows Server 2012R2 uses an extension to the TLS SSL protocol called Server Name Indication – SNI (RFC 4366). This extension allows web servers to present host names when handshaking SSL, so that multiple SSL sites can be hosted on a shared IP-address and port (443) – just like the concept of host headers in good old http have always allowed us to do. This is great because it allows preservation of IP-addresses which also can reduce a total cost of ownership for a service (public IP-addresses does not come for free these days).

What is not so great is that the SNI extension is not yet supported by “everything else in the world” – such as very old web browsers – and Hardware Load Balancers. Using a HLB that does not have a clue about SNI will not work, and that is the topic of this blog post.

If you are interested to read more about SNI in Windows Server 2012R2 – and how to solve issues that can occur, I can recommend Ian Parramore’s two excellent blog post on TechNet, which also have been a great inspiration and source of content to this post, found here, and here.

HOW DO I KNOW THAT MY HLB DOES NOT SUPPORT SNI?

If your HLB does not support SNI, you will notice it eventually when attempting to configure a load balancing and you don’t get any response on port 443 from any of the server IP-addresses. (that’s usually when you spend a day or two troubleshooting firewall ACL’s and routing without any luck). My best advice is to look for SNI-related settings in the HLB administrative interface and read through the release notes for the software version you are running on the appliance(s).

Examples of observed behaviour when attempting to configure HLB with a non-SNI compliant device is a “Http/1.1 Service Unavailable” message from a Citrix NetScaler device and a “Network Error (tcp_error) A communication error occured: Connection refused” message from a BlueCoat device.

Big-IP/F5 customers may turn to a technical article written by Greg Coward for assistance in configuring the Big-IP to work with ADFS 3.0/WAP, found here.

MY HLB DOES NOT SUPPORT SNI – WHAT CAN I DO?

CHECK FOR UPDATES: Before attempting to fix the issue on the Windows Server side – you should always see if there is an available software/firmware update for your HLB device that includes SNI support. There are two workarounds available (see below), but having native support is usually preferred as it reduces the overall complexity of your solution.

WORKAROUND #1 : ADD A HTTPS (PORT 80) BASED HEALTH CHECK PROBE

NOTE: This functionality was made available with the August 2014 Update Rollup for Windows Server 2012R2, found here. Running at least this update rollup level on your ADFS/WAP servers is required for the following workaround to work.

By adding a http-binding on the ADFS/WAP server, your HLB can do the health check probing using the http protocol and thus bypassing SSL and SNI completely. The http endpoint will respond with HTTP 200 OK if available if queried to http://servername-or-ip-address/adfs/probe, and will not automatically allow your ADFS traffic to use HTTP, they will function just as health check probes. Follow these steps to create such binding:

  1. Make sure you have installed all available Windows Updates on the ADFS/WAP machine – at least August 2014 Rollup.
  2. Launch the Windows Firewall with Advanced Security MMC on the first WAP server
  3. Go to Inbound Rules
  4. Create a new Inbound Rule (New Rule in Action Pane). Suggested name: “ADFS HTTP Health Check Probe”
  5. Configure the rule for TCP protocol, local port 80 (specific port) and Allow traffic (All ports as Remote port). See overview of expected result in this picture from Ian Parramore’s blog.
  6. Repeat steps on other ADFS/WAP machines.
  7. Configure HLB to do health check probe using the http protocol and this specific endpoint: http://servername-or-ip-address/adfs/probe

WORKAROUND #2 (RECOMMENDED): ADD A FALLBACK BINDING TO 0.0.0.0:443

It is possible to create a binding to make ADFS 3.0 and/or WAP to respond to SSL requests made out to ip-adresses. This is the recommended workaround as you can’t fully disable the SNI functionality completely. The process of adding this fallback binding consist of finding out the certificate thumbprint of your desired SSL certificate (added with private key to the Local Computer\Personal store) and then binding that thumbprint to all ip-adresses and for the ADFS or WAP application.

Follow these steps on all your ADFS 3.0 servers to add the fallback binding (and make your non-SNI compliant HLB be able to see your ADFS servers):

  1. Make sure that you have installed all available updates for Windows Server 2012R2 after adding and configured the ADFS STS or WAP Proxy role.
  2. Open a Command Prompt as administrator
  3. Run the following command:
    netsh http show sslcert
  4. You will see a list of SSL Certificate bindings.
  5. Mark and copy the ‘Certificate Hash’ value. Paste it to a temporary place (i.e Notepad)
  6. Mark and copy the ‘Application ID’ value. Paste it to the temporary place. The Application ID is what will associate the binding with ADFS 3.0 (for the internal STS servers) and WAP (for the ADFS Proxy).
  7. Now run the following commmand, where you insert the noted ‘Certificate Hash’ and ‘Application ID’ values from above (keep the { } characters):
    netsh http add sslcert ipport=0.0.0.0:443 certhash=Insert_Certificate_Hash_Here appid={Insert_Application_ID_here}
  8. Restart machine and repeat steps for remaining ADFS 3.0 machines.

After adding the fallback binding, your HLB should be able to reach your ADFS servers normally, and you can continue to configure and fine tune your load balancing. Be aware that if you replace your certificate in the future (when it expires), you must perform these steps again (a new certificate means a new certificate hash) – worth mentioning in your system documentation.

SUMMARY AND DISCLAIMER

Adding a http health check probe or a fallback binding can help us fix our load balancing, but my hope is that SNI support in HLB devices will become more of a standard than it is today. Like stated in the article, looking for native support with a software/firmware update and proper documentation from your HLB vendor is always recommended before applying the fallback binding workaround. If you are aware of SNI-related knowledge base articles and software updates from HLB vendors, please share them with me in the comments field below or on Twitter (@JesperStahle) and I will add them to this article.

Shared Accounts (POS/Kiosk) and Single Sign-On in the Same Environment

BACKGROUND

You have successfully configured Single Sign-on in Office 365 with the following conditions:

  1. ADFS is deployed with one internal pair (STS) and one external pair (Proxy), both pairs are load balanced. ADFS is running at least version 2.0 with the latest Update Rollup installed.
  2. Clients that resides on the internal network have the ADFS URL (in this example sts.Company.com) pointing to the ADFS STS server pair (the LB VIP).
  3. Clients that resides on the external network have the ADFS URL (sts.Company.com) pointing to the ADFS Proxy pair (the LB VIP).
  4. On all Active Directory-domain joined clients, the ADFS URL (sts.Company.com) is added to the Local Intranet Zone using a Site-to-Zone Assignment List GPO.
  5. You have not configured ADFS to allow Single Sign-on for non-IE browsers (Disable Extended Protection).
  6. You have verified that Single Sign-on works as expected by navigating to http://outlook.com/OWA/company.com (where Company.com is a custom domain that is configured as a Federated Domain in your Office 365 tenant). When navigating to the URL, you are redirected to Outlook Web App without being prompted for credentials.

This scenario is exactly what we want for our users that are logging on to Windows everyday using their personal Active Directory account. It is a seamless solution that allow the user to consume Office 365 services without entering credentials multiple times through-out the working day.

The problem is if parts of the organization is using common/shared Active Directory accounts for logging on to Windows. A few examples of when common Active Directory accounts are used for logging on to Windows are at: Point of Sale terminals (POS), front desk/receptions, and at heavy industry operation sites (factory users are restricted by gloves/outfit).

These users will not be able to access their personal Office 365 services as they are logged in to Windows with the shared account, and when attempting to access, for example, Outlook Web App, they either get a message that the current account does not have a valid license (if the shared account is not licensed), or they will be logged in to the wrong mailbox (if a licensed user is logged on to Windows when the other user is taking over the session). The user have no chance to login as themselves to reach their personal mailbox etc, as we have configured Single Sign-on to Office 365, without any credential prompts allowed.

In summary, we have a situation where users logging on with personal Active Director accounts to their personal computers have a perfect experience with Single Sign-on, but our users without personal computes that are logging on to Windows with shared accounts, are simply not able to reach their mailbox and other data without first logging in to Windows with their personal account – which is not feasible for the business operation.

CAUSE

This bevaviour is fully expected. It is not ideal for users to share Active Directory accounts for loggin on to Windows, both from the security perspective (complicance and auditing) and from the functional perspective (single sign-on and personal data access). To remediate it, the general recommendation is of course to have all users logging on to Windows with personal Active Directory accounts, but in reality, this is not always possible. The reasons are many, and they are not described on TechNet or in any text book examples, but they are usually related to the time it actually takes to log off and log back on. At a POS, every lost second can be a lost sale, and at the factory floor, every lost second can be a compromise of a persons safety.

Technically, our Single Sign-on configuration is doing exactly what it is supposed to do. A user attempts to reach Outlook Web App, get’s redirected to ADFS STS and authenticates using the Kerberos TGT-ticket issued for the shared account when logging on to Windows. ADFS grants a Token, including claims for the shared account. The Token is then presented to Office 365, who translates the claims to either a non-licensed user (giving the No license error), or if it exist, to the mailbox for the shared account (loading the wrong mailbox for the user).

RESOLUTION

To remediate this problem, without affecting the Single Sign-on experience for our Office workers with personal computers, we must break the Single Sign-on configuration only for the shared accounts.

To accomplish this, we will configure the Site to Zone Assignment List GPO.

As stated in the Background above, the ADFS URL sts.Company.com is added to the Local Intranet Zone to allow Single Sign-on. If we remove it from there, the user will not be able to present the Kerberos TGT-ticket when logging on, and we will prompted for credentials again. This is a bad user experience and should never be the routine for Office users with personal computer and personal accounts, but such credential prompt is what can allow a user that has logged on with a shared user account to actually login to Outlook Web App (or other Office 365 Service) with a personal account and reach the personal mailbox.

High-level description of the solution:

  1. Depending on how Group Policy is used in your organization today, first decide if this setting should be computer based or user based. Both can be applied for this.
  2. Configure separate Site to Zone Assignment List GPO’s for personal Active Directory accounts and for common/shared Active Directory accounts. (Or for personal computers or shared computers, depending of outcome in #1).
  3. For the personal account/computer GPO, place sts.Company.com in the Local Intranet Zone – this will give them Single Sign-on
  4. For the shared accounts/computers, Place sts.Company.com in the Trusted Sites zone – this will break Single Sign-on and give the user a credential prompt on access to the services. The user can respond to the credential prompt with personal credentials and by that accessing the personal mailbox/data.

Here is a good overview of the Site to Zone Assignment List GPO: http://blogs.msdn.com/b/askie/archive/2012/06/05/how-to-configure-internet-explorer-security-zone-sites-using-group-polices.aspx (Scroll down a bit).

COMMENT: Some organizations solves this issue by letting all clients redirect sts.company.com to the ADFS Proxy even internally. This methdo will also present the credential prompt, but it will affect all users – including those that are using personal accounts on personal computers – which is a bad user experience for them. Using GPO and Site to Zone assignment is  a better way to go.

Autodiscover and rich Outlook configuration fails but SSO for OWA/Lync/Portal works (“The AD FS 2.0 Windows Service service failed to start due to the following error: The service did not respond to the start or control request in a timely fashion.” ExRCA: “A network error occurred while communicating with the remote host. “) – CRL check in ADFS is failing

BACKGROUND

The following works:

  • You have successfully deployed ADFS and Single Sign-on  with Office 365
  • You can successfully log on to the Office 365 Portal, Outlook Web App and the rich Lync client using SSO (Active Directory credentials) both from the inside and outside (through ADFS Proxy)
  • You have added the correct CNAME for the Autodiscover service in the public and internal DNS zone for you custom domain
  • You can successfully access the ADFS metadata from the outside by opening https://sts.domain.com/adfs/services/trust/mex in the web browser – it displays content. (sts.domain.com is the URL to your ADFS Proxy)

The following does not work:

  • When attempting to configure an ActiveSync device with Autodiscover (entering e-mail address and AD-password), the configuration fails and the device asks for server name.
  • When attempting to configure your rich Outlook client, the configuration fails and repeatedly ask for username and password during Autodiscover configuration
  • ExRCA (http://testexchangeconnectivity.com) tests for Autodiscover fails with the following error:
    Testing TCP port 443 on host autodiscover.domain.com to ensure it’s listening and open.
      The specified port is either blocked, not listening, or not producing the expected response
    Additional Details
      A network error occurred while communicating with the remote host.

CAUSE

As SSO with the Office 365 portal, Outlook Web App and the rich Lync Client works, as well as accessing the ADFS Metadata, you will probably not think that this issue is ADFS related, but rather related to a back-end error of sort in the Exchange Online service. However, looking in to the ADFS configuration, you may find event 7000 with the following explanation:

The AD FS 2.0 Windows Service service failed to start due to the following error: The service did not respond to the start or control request in a timely fashion.

This is related to why Autodiscover, ActiveSync and the rich Outlook client configuration will not work.

If you investigate the network traffic while attempting to start the ADFS Service, you might find that the service is attempting to do a CRL check for the certificates. This fails, which causes the service start to time out – leaving you with an ADFS service that works good for Office 365 portal, OWA and Lync – but not Autodiscover, ActiveSync and rich Outlook.

This may occur if:

  1. The ADFS servers can’t access the Internet for CRL lookups
  2. You are attempting to use web Proxy for the CRL lookups, but the Proxy configuration is faulty (bad IE Connections settings or missing connectivity between ADFS servers and the web Proxy itself)

The failing CRL check will not reveal itself in the event log, you can however see it by monitoring the network traffic.

RESOLUTION

First, verify that the ADFS Service account have Log on as a Service and Log on as a batch job rights. (The behavior can occur if this is missing as well). Then, follow one of the solutions below to remediate the problem with CRL verification access from the ADFS servers.

Solution 1 – Fix the CRL Access:

  1. Verify that you have proper web Proxy (IE Connections) settings on the ADFS servers. Verify that you can access the Internet, or needed Internet hosts for CRL checks in your Environment.
  2. Start the service and see that it comes online properly.

Solution 2 – Turn off Automatic Root Certificates Update:

  1. Open gpedit.msc (Start\Run)
  2. Navigate to Computer Configuration \ Administrative Templates \ System \ Internet Communication \ Internet Communication
  3. Open Turn off Automatic Root Certificates Update and set it to Enabled
  4. Start the service and see that it comes online properly.

Note: Solution 2 is here described for the local Group Policy which applies if you are using Security Baseline configuration on the server. If an Active Directory GPO is controlling this setting, you need to set this in the AD GPO.

 

After applying Solution 1 or 2 above, and verifying the Log on as a service and Log on as a batch job rights for the ADFS Service Account, Autodiscover and rich Outlook configuration will work.