Author: Josh Bryant

Home / Author: Josh Bryant

Back in April of 2016, Zaid Arafeh, Clare Kearney, and I, recorded a 7 part series for the Microsoft Virtual Academy titled “Defending Active Directory against Cyberattacks”. Unfortunately Microsoft retired the Virtual Academy and most of the recordings are no longer available. Some of the content is now part of this edX course on Managing Identity.

I recently had someone reach out to me on Twitter and ask if I still had a copy of the slides from this series. After some searching, I was able to locate them. Slides for all 7 sessions are now available here below:

Hunting Webshells: Tracking TwoFace

September 9, 2019 | Conference Talks | No Comments

Microsoft Exchange Servers are a high-value target for many adversaries, which makes the investigation of them during Incident Response vital. Backdoor implants in the form of webshells and IIS modules on servers are on the rise. Find out how to hunt webshells and differentiate between legitimate use and attacker activity, using default logging available on every exchange server. The presentation will feature real-world examples carried out by an adversary group using web-based backdoors to breach and maintain access to networks of targeted organizations in the Middle East.

This talk has evolved over the last few years. I originally came up with the idea for this talk in 2016 after some friends and colleagues of mine from Microsoft’s Global Incident Response team (now called Detection and Response Team) reached out to me for help analyzing a webshell they’d found at a customer. By this point I had analyzed a few different webshells targeting Exchange Servers, and thus the original version of this talk, titled “Hunting Webshells on Microsoft Exchange Server” was born and delivered at the 2017 SANS Threat Hunting Summit. At the time I had no idea what to call this webshell.

Shortly after SANS released the recording of that talk, Palo Alto’s Unit 42 released a blog post by Robert Falcone, who called this webshell “TwoFace”. Robert and I started talking and decided to combine our research and co-present the next version of this talk. It was accepted by and delivered at the 2018 SANS Threat Hunting Summit. Video and slides from that version can be found below.

In 2019, we decided to update this talk again. We ended up delivering it at DerbyCon 9: Finish Line. Video and slides from this version can be found below as well.

If you would like this talk presented at your event or conference, please contact me on Twitter or LinkedIn.

In the physical realm, a successful hunt ends with either a kill or a capture.  While some might enjoy the thrill of the hunt, no one really wants to walk away empty handed.  Why do we treat hunting in the digital realm differently?  The Containment, Eradication, and Recovery phase of the Incident Response Lifecycle is the digital equivalent of the kill or capture in the physical world.  Proper execution of this phase is necessary for a successful hunt, and it’s as easy as remembering your ABCs.  You’ve stalked and located your prey, evil has been found, are you prepared to take it out? 

I gave this talk at the GoSec 2019 conference in Montreal, the 2019 Texas Cyber Summit in San Antonio, and the 2019 Forensik conference in Montreal. Slides are available below. I’d like to give a shout out to Michael Melone, who I believe originally came up with the “ABCs”, Jared Poeppleman for writing the krbtgt reset script, as well as Andrew Robbins and his team for creating the Bloodhound tool, all of which are referenced in these slides.

If you are interested in having this talk delivered at your event or conference, please contact me through Twitter or LinkedIn.

Public and private enterprises face the same threats, and yet often have different approaches to defense. What if you could take some of the best tactics from each and blend them together? What if I told you this is already happening in small pockets around the US? Enhance your defenses by studying the strengths and weaknesses of each sector and blending tactics from both.

This talk was original presented at the 2019 SANS Enterprise Defense Summit. Slides are available below.

If you are interested in having this talk delivered at your event or conference, please contact me through Twitter or LinkedIn.

The results are in, you’ve been breached. It’s officially the worst day of your career. How will you handle what comes next? Are you prepared to navigate the long road to recovery? Where do you even begin? Come, hitch a ride with me, I’ll show you the way. Lessons learned from dozens of compromise recoveries across a variety of industries from around the world. Advice on evicting your adversary, answering to Executives, and recovering from the trauma of a cyberattack, to help you better prepare for the inevitable breach. Turn your worst day around, just don’t forget your towel!

This talk was given at the 2017 SANS Data Breach Summit, and the 2018 Camp IT Enterprise Risk / Security Management conference to audiences of Executives and Legal Council. Recently a few people expressed interest in seeing the slide deck from this talk, so I’m making it available here.

If you are interested in having this talk delivered at your event or conference, please contact me through Twitter or LinkedIn.

Three years ago I wrote a document titled “Removing Exchange’s ability to impact Tier 0 and Tier 1” that was distributed internally at Microsoft as well as to dozens of Microsoft customers as part of Cybersecurity services delivered through Microsoft Consulting Services (MCS).

I had always intended to get this document published publicly, but for one reason or another, it never happened. With KB4490059 being released in response to @_dirkjan‘s #PriveExchange vulnerability, I’ve decided it’s beyond time to just share it here on my blog. KB4490059 is a step in the right direction. You can add these mitigations on top of it for even further protections, or perhaps find a more compelling reason to switch to the Active Directory Split Permissions model.

Removing Exchange’s ability to impact Tier 0 and Tier 1

Exchange is tightly integrated with Active Directory. When Exchange is installed, the Active Directory Schema is extended, and several highly privileged groups are created. These privileged groups are granted rights to a large number of objects within Active Directory.  A compromised Exchange Server can easily lead to complete control of Tier 1, and in many instances Tier 0.

For more information on the Tier model, see “Securing Privileged Access”.

In Exchange 2010 and later, there are two options for mitigating this risk. Both should be evaluated carefully before implementation. It is only necessary to implement one of these two options.

Mitigation Option 1 – Implement the Active Directory Split Permissions Model

Starting with Exchange 2010, there are three Permissions Models available. Of the three, the Active Directory Split Permissions Model, is the only one that can mitigate this risk without additional modifications to Active Directory. Because this model does not allow Exchange to modify security principles (i.e. create/delete/reset password) or group membership (i.e. add members to security groups that may grant access to other systems), compromise of an Exchange Server is contained to the Exchange environment when running under this permissions model.

This permissions model isn’t for everyone. You should carefully consider the impact switching to Active Directory Split Permissions may have on your organization. It may require a significant change in user provisioning, business practices, or workflows that revolve around Exchange. It is also important to note that while this permissions model is fully supported by the Exchange Product Group, it does not receive the same level of testing as the default Shared Permissions model.

If you’ve determined that the Active Directory Split Permissions Model is right for your organization, you can switch to it by running the following setup command:

setup.com /PrepareAD /ActiveDirectorySplitPermissions:true

For more information on configuring split permissions, see “Configure Exchange 2010 for Split Permissions”.

Implementing the Active Directory Split Permissions model both removes Exchange from Tier 0, placing it in Tier 1, and reduces potential impact to Tier 1.

Mitigation Option 2 – Reduce privileges granted to Exchange in Active Directory

If your organization needs to use the default Shared Permissions Model, the RBAC Split Permissions Model, or still has legacy versions of Exchange present, modifications to permissions in Active Directory are required to mitigate this risk.

To remove Exchange’s potential to impact to Tier 0, permissions granted to Exchange’s privileged groups must be removed from the Access Control List (ACL) on the following groups and their members in all domains within the Active Directory Forest Exchange has been installed in, to include any nested groups and members of those nested groups.

  • Enterprise Admins
  • Domain Admins
  • Schema Admin
  • BUILTIN\Administrators
  • Account Operators
  • Backup Operators
  • Print Operators
  • Server Operators
  • Domain Controllers
  • Read-only Domain Controllers
  • Group Policy Creators Owners
  • Cryptographic Operators
  • Other Delegated Groups

With the exception of the last 3 entries above, these groups should already be protected by AdminSDHolder, which prevents them from inheriting permissions that would allow Exchange’s privileged groups (Exchange Trusted Subsystem, Exchange Windows Permissions, Organization Management, etc.) to take control of them. Members of these groups should have the same protection. However, since Account Operators, Backup Operators, Print Operators, and Server Operators, can be excluded from AdminSDHolder protection, it is worthwhile to take the extra step of removing Exchange’s permission to all of the above groups (if present).

Because the permissions are inherited, the easiest way to accomplish this, is to place all Tier 0 users and groups into the same OU, block inheritance on that OU, and restrict permissions on the ACL for that OU. Full Control (GenericAll), Write (GenericWrite), Modify Permissions (WriteDacl), Write Member (WriteMember), and Change Password (note that Change Password permissions should NOT be removed from “Everyone” or “SELF”), are the permissions that should be removed from any object that should not have the ability to manipulate Tier 0 OUs/users/groups. In most cases the entire security principal can be removed from the Tier 0 OUs/users/groups.

To remove Exchange’s potential to impact Tier 1, a similar approach to the above must be taken.  Exchange’s permissions on the ACLs of users and groups with privileged access to Tier 1 assets.

Because the permissions are inherited, the easiest way to accomplish this, is to place all Tier 1 privileged users and groups into the same OU, block inheritance on that OU, and restrict permissions on the ACL for that OU. Full Control (GenericAll), Write (GenericWrite), Modify Permissions (WriteDACLl), and Write Member (WriteMember), are the permissions that should be removed from any object that should not have the ability to manipulate Tier 1 privileged users/groups. In most cases the entire security principal can be removed from the ACL on the Tier 1 OUs/users/groups.

For example, Exchange should not have permissions to an account or group that grants privileged access to a SQL server.

Privileged accounts and groups should NOT be mail enabled. Therefore removing Exchange’s permissions from them should have no negative impact on operations.  If an account with, or a group used to grant privileged access to, any system, regardless of what tier it resides in, has been mail enabled, privileges should either by removed, or they should be mail disabled. Either way, it may be necessary to create unprivileged accounts/groups for mail use, or non-mail enabled accounts/groups for privileged access use, in order to avoid disruptions.

This approach is not tested by the Exchange Product Group. Failure to exercise caution when implementing this approach could result in damage to the Exchange environment.

Additional Considerations

Service Packs and Cumulative Updates in Exchange often include Schema updates to Active Directory. Because Schema extensions and Active Directory preparation require Schema Admins and Enterprise Admins privileges, this portion of updating Exchange is a Tier 0 operation. Schema extensions and Active Directory preparation must be performed from a Tier 0 Privileged Access Workstation (PAW).

Schema extension and Active Directory preparation performed during an update to Exchange may overwrite some Exchange related settings in Active Directory. It is recommended that permissions on Tier 0 and Tier 1 set according to the guidance above are validated after every Exchange update install.

How fitting that 7 years after I started this blog I would relaunch it on a new platform (more on that to come). I know there’s been a lot of times it’s been neglected over the years due to my work/travel schedule, but that’s changing now.  I’d like to thank anyone and everyone who’s stuck with me reading it over the years.  I’ve got a bunch of new stuff planned, so stay tuned!

Microsoft Exchange Servers are a high value target, making investigation of them during Incident Response vital, but where do you start? What should you look for? Backdoor implants in the form of webshells hiding in OWA are on the rise. Find out how to hunt webshells and differentiate between legitimate use and attacker activity, using default logging available on every Exchange Server, through real world examples. It’s easier than you might think, and these techniques can help up your DFIR game in environments containing Exchange Servers!

This talk was originally presented at the 2017 SANS Threat Hunting and Incident Response Summit. A recording of this talk and the slide deck from it are available below. If you would like an updated version of this talk presented at your event or conference, please contact me on Twitter or LinkedIn.

I’ll be presenting a brand new session titled “Hunting Webshells on Microsoft Exchange Server” at the 2017 SANS Threat Hunting and Incident Response Summit in New Orleans on April 18th and 19th!

My session abstract:
“Microsoft Exchange Servers are a high value target, making investigation of them during Incident Response vital, but where do you start? What should you look for? Backdoor implants in the form of webshells hiding in OWA are on the rise. Find out how to hunt webshells and differentiate between legitimate use and attacker activity, using default logging available on every Exchange Server, through real world examples. It’s easier than you might think, and these techniques can help up your DFIR game in environments containing Exchange Servers!”

Full agenda: https://www.sans.org/event-downloads/45247/agenda.pdf

More details: https://www.sans.org/event/threat-hunting-and-incident-response-summit-2017

If you can’t make the summit, a recording should be available afterwards.  I’ll post a link to the recording and a detailed blog on this subject when available.

Exchange 2016 SP1 to run on Linux!

April 1, 2016 | Uncategorized | No Comments

Ever since last month’s announcement that Microsoft SQL Server will be coming to Linux, quiet rumors have been floating around that some of Microsoft’s other Enterprise Products, such as Exchange Server, may follow suit. With this week’s announcement at the Build conference about the popular Linux shell “BASH” coming to Windows, I decided it was time to see what the Exchange team has planned for Linux, if anything. Today I caught up with a member of the Exchange team that wishes to remain anonymous to get the inside scoop on Exchange 2016 SP1 and support for installing it on Linux!

FixTheExchange: Are there any plans to make Exchange run on Linux?

Microsoft Spokesperson: Originally, no.  We started thinking about it after the Azure team released the Azure Cloud Switch, which is based entirely on Linux. Then when the SQL team announced their product would run on Linux by mid-2017, we knew we had to take the idea more seriously.

FixTheExchange: Does this mean Exchange will run on SQL when it arrives on Linux?

Microsoft Spokesperson: No! Why does everyone keep asking if Exchange will run on SQL?  SQL can’t keep up with us! It’ll still run on ESE, we’ll just be bringing ESE over to Linux along with the rest of our code.

FixTheExchange: Why Linux?

Microsoft Spokesperson: If you think about it, it makes sense. The Office team has made Outlook and the rest of the Office Suite available across several platforms. Outlook on the Web works on browsers across multiple platforms. If our client side is cross-platform, why shouldn’t the servers they connect to them be?

FixTheExchange: When will Exchange be available on Linux?

Microsoft Spokesperson: When it’s ready. 😉 Just kidding, it’s already there. We’ve started testing it in Office 365, just like we would any other new feature for Exchange.

FixTheExchange: Wow, that was quick! What about On-premises?

Microsoft Spokesperson: We couldn’t let the SQL team beat us to it. Our ability to roll changes out quickly in Office 365 really gives us the advantage. We’ll have it ready for on-premises by Service Pack 1, so 4th quarter of 2016.

FixTheExchange: Will there be support for mixing Exchange running on Windows and Exchange running on Linux in the same DAG?

Microsoft Spokesperson: Seriously? Do you think we’re some sort of fools? No. We don’t support mixing versions of Windows in the same DAG, why would we support completely different operating systems within the same DAG?

FixTheExchange: You know it had to be asked… How will Exchange be administered when running on Linux?

Microsoft Spokesperson: We’re working on porting the Exchange Management Shell over to BASH. With the Windows team adopting BASH, you should be able to administer Exchange from either a Windows or Linux desktop.

FixTheExchange: What about PowerShell?

Microsoft Spokesperson: We’re not abandoning PowerShell, just giving administrators more options.

FixTheExchange: Looks like we’re out of time… thanks for sharing this exciting information!

 

You heard it here first folks, not to be outdone by the SQL team, Exchange will reach Linux first!