This talk has evolved over the last few years. I originally came up with the idea for this talk in 2016 after some friends and colleagues of mine from Microsoft’s Global Incident Response team (now called Detection and Response Team) reached out to me for help analyzing a webshell they’d found at a customer. By this point I had analyzed a few different webshells targeting Exchange Servers, and thus the original version of this talk, titled “Hunting Webshells on Microsoft Exchange Server” was born and delivered at the 2017 SANS Threat Hunting Summit. At the time I had no idea what to call this webshell.

Shortly after SANS released the recording of that talk, Palo Alto’s Unit 42 released a blog post by Robert Falcone, who called this webshell “TwoFace”. Robert and I started talking and decided to combine our research and co-present the next version of this talk. It was accepted by and delivered at the 2018 SANS Threat Hunting Summit. Video and slides from that version can be found below.

In 2019, we decided to update this talk again. We ended up delivering it at DerbyCon 9: Finish Line. Video and slides from this version can be found below as well.

If you would like this talk presented at your event or conference, please contact me on Twitter or LinkedIn.

Josh Bryant

Cybersecurity Product Leader and internationally recognized Speaker at several Information Security conferences and events with over 26 years of experience as a proven leader in Cybersecurity, Product Management, Threat Hunting, Incident Response, IT Management, IT Architecture, IT Operations, IT Engineering, and Messaging Systems, on a global scale, across a diverse set of industries in both the Public and Private sectors.

192.168.2.21:30040

I had always intended to get this document published publicly, but for one reason or another, it never happened. With KB4490059 being released in response to @_dirkjan‘s #PriveExchange vulnerability, I’ve decided it’s beyond time to just share it here on my blog. KB4490059 is a step in the right direction. You can add these mitigations on top of it for even further protections, or perhaps find a more compelling reason to switch to the Active Directory Split Permissions model.

Removing Exchange’s ability to impact Tier 0 and Tier 1

Exchange is tightly integrated with Active Directory. When Exchange is installed, the Active Directory Schema is extended, and several highly privileged groups are created. These privileged groups are granted rights to a large number of objects within Active Directory.  A compromised Exchange Server can easily lead to complete control of Tier 1, and in many instances Tier 0.

For more information on the Tier model, see “Securing Privileged Access”.

In Exchange 2010 and later, there are two options for mitigating this risk. Both should be evaluated carefully before implementation. It is only necessary to implement one of these two options.

Mitigation Option 1 – Implement the Active Directory Split Permissions Model

Starting with Exchange 2010, there are three Permissions Models available. Of the three, the Active Directory Split Permissions Model, is the only one that can mitigate this risk without additional modifications to Active Directory. Because this model does not allow Exchange to modify security principles (i.e. create/delete/reset password) or group membership (i.e. add members to security groups that may grant access to other systems), compromise of an Exchange Server is contained to the Exchange environment when running under this permissions model.

This permissions model isn’t for everyone. You should carefully consider the impact switching to Active Directory Split Permissions may have on your organization. It may require a significant change in user provisioning, business practices, or workflows that revolve around Exchange. It is also important to note that while this permissions model is fully supported by the Exchange Product Group, it does not receive the same level of testing as the default Shared Permissions model.

If you’ve determined that the Active Directory Split Permissions Model is right for your organization, you can switch to it by running the following setup command:

setup.com /PrepareAD /ActiveDirectorySplitPermissions:true

For more information on configuring split permissions, see “Configure Exchange 2010 for Split Permissions”.

Implementing the Active Directory Split Permissions model both removes Exchange from Tier 0, placing it in Tier 1, and reduces potential impact to Tier 1.

Mitigation Option 2 – Reduce privileges granted to Exchange in Active Directory

If your organization needs to use the default Shared Permissions Model, the RBAC Split Permissions Model, or still has legacy versions of Exchange present, modifications to permissions in Active Directory are required to mitigate this risk.

To remove Exchange’s potential to impact to Tier 0, permissions granted to Exchange’s privileged groups must be removed from the Access Control List (ACL) on the following groups and their members in all domains within the Active Directory Forest Exchange has been installed in, to include any nested groups and members of those nested groups.

• Enterprise Admins
• Domain Admins
• Schema Admin
• BUILTIN\Administrators
• Account Operators
• Backup Operators
• Print Operators
• Server Operators
• Domain Controllers
• Read-only Domain Controllers
• Group Policy Creators Owners
• Cryptographic Operators
• Other Delegated Groups

With the exception of the last 3 entries above, these groups should already be protected by AdminSDHolder, which prevents them from inheriting permissions that would allow Exchange’s privileged groups (Exchange Trusted Subsystem, Exchange Windows Permissions, Organization Management, etc.) to take control of them. Members of these groups should have the same protection. However, since Account Operators, Backup Operators, Print Operators, and Server Operators, can be excluded from AdminSDHolder protection, it is worthwhile to take the extra step of removing Exchange’s permission to all of the above groups (if present).

Because the permissions are inherited, the easiest way to accomplish this, is to place all Tier 0 users and groups into the same OU, block inheritance on that OU, and restrict permissions on the ACL for that OU. Full Control (GenericAll), Write (GenericWrite), Modify Permissions (WriteDacl), Write Member (WriteMember), and Change Password (note that Change Password permissions should NOT be removed from “Everyone” or “SELF”), are the permissions that should be removed from any object that should not have the ability to manipulate Tier 0 OUs/users/groups. In most cases the entire security principal can be removed from the Tier 0 OUs/users/groups.

To remove Exchange’s potential to impact Tier 1, a similar approach to the above must be taken.  Exchange’s permissions on the ACLs of users and groups with privileged access to Tier 1 assets.

Because the permissions are inherited, the easiest way to accomplish this, is to place all Tier 1 privileged users and groups into the same OU, block inheritance on that OU, and restrict permissions on the ACL for that OU. Full Control (GenericAll), Write (GenericWrite), Modify Permissions (WriteDACLl), and Write Member (WriteMember), are the permissions that should be removed from any object that should not have the ability to manipulate Tier 1 privileged users/groups. In most cases the entire security principal can be removed from the ACL on the Tier 1 OUs/users/groups.

For example, Exchange should not have permissions to an account or group that grants privileged access to a SQL server.

Privileged accounts and groups should NOT be mail enabled. Therefore removing Exchange’s permissions from them should have no negative impact on operations.  If an account with, or a group used to grant privileged access to, any system, regardless of what tier it resides in, has been mail enabled, privileges should either by removed, or they should be mail disabled. Either way, it may be necessary to create unprivileged accounts/groups for mail use, or non-mail enabled accounts/groups for privileged access use, in order to avoid disruptions.

This approach is not tested by the Exchange Product Group. Failure to exercise caution when implementing this approach could result in damage to the Exchange environment.

Additional Considerations

Service Packs and Cumulative Updates in Exchange often include Schema updates to Active Directory. Because Schema extensions and Active Directory preparation require Schema Admins and Enterprise Admins privileges, this portion of updating Exchange is a Tier 0 operation. Schema extensions and Active Directory preparation must be performed from a Tier 0 Privileged Access Workstation (PAW).

Schema extension and Active Directory preparation performed during an update to Exchange may overwrite some Exchange related settings in Active Directory. It is recommended that permissions on Tier 0 and Tier 1 set according to the guidance above are validated after every Exchange update install.

Josh Bryant

Cybersecurity Product Leader and internationally recognized Speaker at several Information Security conferences and events with over 26 years of experience as a proven leader in Cybersecurity, Product Management, Threat Hunting, Incident Response, IT Management, IT Architecture, IT Operations, IT Engineering, and Messaging Systems, on a global scale, across a diverse set of industries in both the Public and Private sectors.

192.168.2.21:30040

This talk was originally presented at the 2017 SANS Threat Hunting and Incident Response Summit. A recording of this talk and the slide deck from it are available below. If you would like an updated version of this talk presented at your event or conference, please contact me on Twitter or LinkedIn.

Josh Bryant

Cybersecurity Product Leader and internationally recognized Speaker at several Information Security conferences and events with over 26 years of experience as a proven leader in Cybersecurity, Product Management, Threat Hunting, Incident Response, IT Management, IT Architecture, IT Operations, IT Engineering, and Messaging Systems, on a global scale, across a diverse set of industries in both the Public and Private sectors.

192.168.2.21:30040

Symptoms
Outlook Profile creation either fails after a single PIN prompt with a message stating that encrypted communication with the Exchange Server could not be established, or profile creation never progresses past the first stage with repeated PIN prompts.

Additionally, using the “Test E-mail AutoConfiguration” feature in Outlook (CTRL + Right Click on the Outlook icon in the system tray) returns error 0x80090014 on the log tab for Autodiscover. (Note: If no profiles exist on the problem computer, you can open Outlook without creating a profile to access this functionality).​

Cause
0x80090014 = NTE_BAD_PROV_TYPE, “Invalid provider type specified”. This occurs when an application tries to use a Cryptographic Service Provider (CSP) that Windows isn’t aware of. This may be the result of the version of Windows not supporting a CSP, or Smartcard Middleware not properly installing a 3^rd Party CSP required by the certificate on the Smartcard.

Resolution
Verify that the version of Windows you are running supports the CSP used by the certificate on your Smartcard, and that any 3^rd Party CSPs/Middleware required by your certificate are installed and properly functioning.

You can see the available CSPs by viewing the following Registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\Defaults\Providers

In this case, the customer had 3^rd Party (ActivIdentity’s ActivClient) Middleware installed. We found that version 7.x was installed on the Windows 8.1/Outlook 2013 clients that couldn’t connect, and version 6.2.9200 on the Windows 7/Outlook 2010 clients that were able to connect.

We found that the non-functioning clients were missing the following registry key:

HKEY_LOCAL_MACHINE\Software\Microsoft\Cryptography\Defaults\Provider\ActivClient Cryptographic Service Provider

They were also missing a DLL identified by one of the values under that key:
C:\Program Files\ActiveIdentity\ActivClient\accsp.dll

Without that registry key and DLL, Windows couldn’t find the CSP from the certificate we selected on our Smartcard when prompted, which is why we saw the 0x80090014 error.

Updating to the latest 7.x version we could find did not resolve this issue. We had to uninstall 7.x and install 6.2.9200 (WARNING! This requires an account with local administrative privileges that is allowed to logon with username and password! After the required reboot to uninstall the 7.x version, you will not be able to logon with a Smartcard. You must install the working 6.2.9200 version in order to use Smartcard logon again.)

After downgrading the Middleware to a version that included the necessary registry entries and DLL file, the Windows 8.1 w/ Outlook 2013 clients were able to successfully connect to Exchange over Outlook Anywhere with Smartcard Authentication.

Josh Bryant

Cybersecurity Product Leader and internationally recognized Speaker at several Information Security conferences and events with over 26 years of experience as a proven leader in Cybersecurity, Product Management, Threat Hunting, Incident Response, IT Management, IT Architecture, IT Operations, IT Engineering, and Messaging Systems, on a global scale, across a diverse set of industries in both the Public and Private sectors.

192.168.2.21:30040

There have been reports of RU8 breaking Outlook connectivity.  Because of this, RU8 is being recalled, so expect an RU8v2.  For more information, see the Exchange Team blog post that was updated today.

Josh Bryant

Cybersecurity Product Leader and internationally recognized Speaker at several Information Security conferences and events with over 26 years of experience as a proven leader in Cybersecurity, Product Management, Threat Hunting, Incident Response, IT Management, IT Architecture, IT Operations, IT Engineering, and Messaging Systems, on a global scale, across a diverse set of industries in both the Public and Private sectors.

192.168.2.21:30040

Exchange 2010 SP3 RU7 has a handful of bug fixes, and so far is rather unintersting and uneventful.

Exchange 2013 CU6 on the other hand, has had a couple major issues.  The first issue only happens if you're in Co-Existence with Exchange 2007 and have a Database Availabilty Group (DAG).  After installing CU6, Databases in your DAG may failover unexpectedly.  An interim update is available to fix this issue. You'll have to contact Microsoft Support to obtain it if you need it.

The second issue only happens if you're in a Hybrid deployment with Office 365.  CU6 includes the fix for KB2988229 where running/rerunning the Hybrid Configuration Wizard would fail in CU5 or earlier due to a change made on the Office 365 side.  Great news! Except that it breaks some basic functionality for your Hybrid deployment, like creating a mailbox. There's a script to fix this, but the script will fail if you've installed Exchange anywhere BUT the default install path.  You can fix it by changing the baseDirectory in the script (found in 3 spots) to this: $baseDirectory =  "$Env:ExchangeInstallPath" + "ClientAccess\ecp\DDI"

Josh Bryant

Cybersecurity Product Leader and internationally recognized Speaker at several Information Security conferences and events with over 26 years of experience as a proven leader in Cybersecurity, Product Management, Threat Hunting, Incident Response, IT Management, IT Architecture, IT Operations, IT Engineering, and Messaging Systems, on a global scale, across a diverse set of industries in both the Public and Private sectors.

192.168.2.21:30040