Hybrid Email Moderation in Office 365
This article talks about the email moderation in Office 365, when people have mailboxes on-premises and on others on Office 365. In certain cases, the approval process does not work well, if we have mixed of approvals and senders between on-premises and Office 365. Let us go to the basics and see how email moderation really works in Microsoft Exchange.
How Email Moderation Works
To understand hybrid email moderation in Office 365, let us see how moderation generally works:
- Mail user sends an email to a moderated group.
- The categorizer component in the Exchange hub transport server intercepts the email marked for moderation and then re-routes it to the arbitration mailbox.
- The store driver component stores the message in the arbitration mailbox and sends an approval request to the moderator.
- Arbitration mailbox is a system mailbox that get created when you install Microsoft Exchange.
- The moderator takes an action (approve or reject)
- The store driver component marks the moderator’s decision on the original message stored in the arbitration mailbox.
- The Information Assistant component reads the approval status on the message stored in the arbitration mailbox, and then process the message depending on the moderator’s decision.
- If the moderator has approved the message, the Information Assistant component resubmits the message to the submission queue, and the message is delivered to the recipient(s).
- If the moderator has rejected the message, the Information Assistant component deletes the message from the arbitration mailbox and notifies the sender that the message was rejected.
So What are the Arbitration Mailboxes?
Arbitration mailboxes play big role in hybrid email moderation in Office 365 . Arbitration mailboxes are system mailboxes that get created by default in any Exchange organization. They have a well known names that is similar in any Exchange organization.
You can view them by typing Get-Mailbox -Arbitration
The one that we are interesting in here is the first one (Microsoft Exchange Approval Assistant). This is the system mailbox that handles moderation.
IT Admin can create as many arbitration mailboxes for moderation to load balance the approval process, but we choose to stick with this one default one.
Note that in the on-premises Exchange implementation, we know we have only one arbitration mailbox for moderation and we know its address by typing Get-Mailbox - Arbitration .
In Exchange Online, the property -Arbitration does not exist in the Get-Mailbox command. This is because it is hosted environment, and Microsoft seems to have lots of those arbitration mailboxes and they are hidden from us. We just know that some arbitration mailbox in the cloud will handle the approval process, we just know which one and we cannot query it.
We only know and this is so important, that the Exchange Online arbitration mailbox has email address that ends with @contoso.onmicrosoft.com and NOT @contoso.mail.onmicrosoft.com. Strange enough as we all know that contoso.onmirosoft.com is identity domain, while @contoso.mail.onmicrosoft.com is a mail routing domain.
The One Million Dollars Question: Which Arbitration Mailbox will be used?
The big question that we should ask our selves when dealing with hybrid email moderation in Office 365 is to know which arbitration mailbox will be used. So we have people in Office 365 and their mailboxes are in Exchange Online, and we have people on-premises with mailboxes in our on-premises Exchange Environment. Now, which arbitration mailbox will be used when sending to a moderated group? The Exchange Online one or the one we know about in our Exchange on-premises system?
Why this is important is coming soon, but it is so important to understand which arbitration mailbox will be invoked. There are two types of moderated groups on our on-premises Exchange system:
- Dynamic groups that are moderated.
- Non-Dynamic groups that are moderated.
Why this is important? Well, because the way AD Sync works.
Dynamic groups created on-premises are not synchronized to Azure AD. Azure AD does not recognized on-premises dynamic groups. This means that all on-premises dynamic groups are not available in the address book of Office 365 mailbox users.
Trick: To make Office 365 mailbox users see on-premises dynamic groups, you can create exchange contacts in Azure AD with the same name as on-premises dynamic groups, and same email address as the on-premises dynamic groups, assuming that the SMTP domain is set as internal and not authoritative, so that Exchange Online will forward the email to the on-premises Exchange hybrid servers.
So why this is important? Well, the only thing that we want to make sure we understand is that when Office 365 hosted mailbox user is sending email to an on-premises dynamic group, there is no way for Exchange Online to know that this group is moderated because AD Sync does not sync dynamic groups to Azure AD, instead it will route it to our Exchange on-premises servers to deal with it. Now Exchange on-premises knows that this is a moderated group, and it will put that email to the on-premises arbitration mailbox. Sound good?!
While if a an Office 365 hosted mailbox user is sending email to an on-premises non-dynamic group, Exchange Online will query Azure AD, and since this group is synced, Exchange Online at this moment, knows and realize this is a moderated group, because the group ,and all of its properties including moderation information, are synced and available in Azure AD. So Exchange Online will try to behave in a smart way and start taking responsibility. It will not dare to forward the email to Exchange on-premises, as it feels that it should stand to the challenge. It knows it is a moderated group, so it will try to save the day. It will shout (Hey Exchange Online, give me an available arbitration mailbox, I have work to be done), and it will through the email to one of the many available arbitration mailboxes in Exchange Online, that we do not know their names or specific email address. We only dare to know that the email address of those cloud arbitration mailboxes ends with @contoso.onmicrosoft.com (yes not @mail.contoso.onmicrosoft.com)
Before anything start working, there are couple of prerequisites to be accomplished before making moderation works between on-premises and Exchange Online.
- On the outbound connector to Office 365, we shall add @contoso.onmicrosoft.com in the address space for that connector. That is, we need to educate our Exchange on-premises that anything going to @contoso.onmicrosoft.com should be routed to Exchange Online. This is necessary for the approval messages from on-premises moderators to reach the Exchange Online arbitration mailbox (that has email address ends with @contoso.onmicrosoft.com).
- In order for the button called Approve and Reject to work correctly in hybrid moderation, we need to configure remote domains and configure them in both on-premises and Exchange Online. This would be mentioned later in a coming blog post.
How things works?
Well, we have 8 possibilities to take care of when evaluating how hybrid moderation works.
- On-premises user sending to on-premises Dynamic Group and the approval is on-premises user. [We know it will work]
- On-premises user sending to on-premises Dynamic Group and the approval is cloud user.
- Cloud user sending to on-premises Dynamic Group and the approval is on-premises user.
- Cloud user sending to on-premises Dynamic Group and the approval is cloud user.
- On-premises user sending to on-premises Non-Dynamic Group and the approval is on-premises user.[We know it will work]
- On-Premises user sending to on-premises Non-Dynamic Group and the approval is cloud user.
- Cloud user sending to on-premises Non-Dynamic Group and the approval is on-premises.
- Cloud user sending to on-premises Non-Dynamic Group and the approval is cloud user.
- Cloud user sending to cloud user or cloud group, and the approval is on-premises.
We will go through those cases and try to see which arbitration mailbox will be invoked and what to expect. The rule is [THE FIRST EXCHANGE SERVER WHO KNOWS ABOUT MODERATION, WILL USE HIS ARBITRATION MAILBOX]
I assume that your contoso.com domain is set to Internal Relay in Exchange Online, so that if Exchange Online want to send email to one of contoso.com recipient and cannot find a match in Azure AD, it will forward it to your on-premises Exchange server. Of course setting contoso.com as internal relay means that Directory Based Edge Blocking will be disabled for your contoso.com recipients.
Case 2: On-premises user sending to on-premises Dynamic Group and the approval is cloud user.
Since the sender [Let us say Alice] is on premises, and the on-premises Exchange knows about the fact that the dynamic group is moderated, it will use the on-premises arbitration mailbox. So the email will be forwarded to the on-premises arbitration mailbox.
The arbitration mailbox will send email to the moderator [say his name is Bob and his mailbox is in Office 365]. Bob will receive email from [MSExchApproval1f05a927email@example.com] saying “hey please approve this email”.
Bob when he approves the email, he is like sending email to that arbitration mailbox, so it is Bob’s mailbox sending email to MSExchApproval1f05a927firstname.lastname@example.org. All sounds good. Arbitration mailbox will release the email, letting the on-premises Exchange transport engine to deliver the email to that group.
Case 3: Cloud user sending to on-premises Dynamic Group and the approval is on-premises.
Now this is interesting. Alice is Office 365 user and trying to send email to dynamic group on-premises that is moderated. Since Exchange Online does not know anything about Dynamic Groups created on-premises because they are not synced to Azure AD, it will just forward the email as is, to on-premises Exchange servers.
On-premises Exchange will send the email to the on-premises arbitration mailbox, approval will be sent to the on-premises moderator from the on-premises arbitration mailbox and everything works fine here. The on-premises moderator will approve the email, so he will be replying to the on-premises arbitration mailbox, and then the transport engine will release the message.
Case 4: Cloud user sending to on-premises Dynamic Group and the approval is cloud user.
This is similar to the previous case. the on-premises arbitration mailbox will be used as the Exchange Online system does not know that the group is moderated as it is not synced to Azure AD. Of course now the on-premises arbitration mailbox will send approval email to the cloud moderator. The cloud moderator will see the email coming from the on-premises moderation mailbox, he will reply back by approving or rejecting, and all will work just fine.
Case 6: On-premises sending to on-Premises Non-Dynamic Group and the approval is cloud user.
Alice is on-premises user, trying to send email to a non-dynamic group on-premises, so the on-premises Exchange will pickup the message, and since it is aware that the group is moderated, it will send the message to the on-premises arbitration mailbox. Nothing interesting here and everything will work just fine.
Case 7: Cloud user sending to on-premises Non-Dynamic Group and the approval is on-premises.
THIS IS SO IMPORTANT CASE: IT WILL BREAK AND NOT WORK !!!
So now Alice is Office 365 user trying to send email to on-premises non-dynamic moderated group. Since Alice is in Office 365, Exchange Online is the first point of contact here, and it knows, yes it knows, that that on-premises group is moderated. Why? because it is synced to Azure AD, in contrast to the dynamic groups which are not synced.
So here we go!. Exchange Online will send the email to the Exchange Online arbitration mailbox. Remember the rule: [THE FIRST EXCHANGE SERVER WHO KNOWS ABOUT MODERATION, WILL USE HIS ARBITRATION MAILBOX].
We only know that the Exchange Online arbitration mailbox has an email address that ends with @contoso.onmicrosoft.com.
Now the arbitration mailbox in Office 365 will send email to Bob the moderator whose mailbox is located on the on-premises Exchange system. The email looks like this:
Please approve or reject this message from Alice to this group.
Bob is a good dude, we will click APPROVE. and let us pause for a second here. Bob by pressing APPROVE, he is like sending email that looks like this:
Please approve or reject this message from Alice to this group.
Guess who will be sending this message? Of course Exchange on-premises, and since there is a connector between on-premises Exchange and Exchange Online with @contoso.onmicrosoft.com as a destination SMTP address space, the hybrid servers will forward the message to Exchange Online.
Guess who is protecting Exchange Online? Yes it is Exchange Online Protection EOP. This EOP system has a wall of defense called Accepted Domains and Directory Based Edge Blocking. Also the domain contoso.onmicrosoft.com is defined as Authoritative domain in Exchange Online.
So what all that means? Well, in Exchange Online, Accepted Domains are the SMTP domains that EOP receives email to. When a domain is configured as Authoritative Accepted Domain, EOP applies a firewall defense called Dictionary Based Edge Blocking, that does the following:
- If I receive an email coming to an authoritative domain, the recipient in that email should be well defined as a recipient in the Azure AD [Exchange Online Address Book].
- Else, drop the email.
So what is the point here? This is to protect from dictionary attack, where someone tries to open a dictionary and tries every possible first and last name, and tries to send storm of possible smtp addresses for a domain. If this dictionary based edge blocking is not in place, then EOP would send all those attempts to the internal Exchange system to deal with, and filter wrong address from right ones, which consume the resources of the internal email system.
To return back and talk about our case, now the email is coming from on-premises Exchange from Bob@contoso.com and going to some GUID address that ends with @contoso.onmicrosoft.com via the outbound connector to Office 365. Now EOP will check the domain @contoso.onmicrosoft.com and it will see it configured as Authoritative domain, and will apply the Dictionary Based Edge Blocking knowledge. So it will look at the Azure AD and will ask Azure AD ” Hey Azure AD, do you have a recipient with email address StrangeGuid@contoso.onmicrosoft.com”. Azure AD will say “hell, no”, and EOP will block that message. Interesting right? This took us long time to figure it out. Solution will be mentioned later in this blog post.
Case 8: Cloud user sending to on-premises Non-Dynamic Group and the approval is cloud user.
This is straight forward, so the Exchange Online arbitration will pickup the message, and communicate with the cloud moderator, and all will work just fine. No traffic will reach the on-premises Exchange environment until the final group delivery in case the group contains on-premises recipients.
Case 9: Cloud user sending to cloud user or cloud group and the approval is on-premises.
Since Exchange Online knows that the recipient user or group is moderated, then the system mailbox of Exchange Online will kickoff and will send email to the on-premises moderator. When the on-premises moderator tries to approve the message, he will be sending an email to the Exchange Online system mailbox, which will not pass by. This is somehow similar to case 7. The best way to solve this is by migrated the mailbox of the moderator to Exchange Online.
Case 7 Solution:
Why you should not Dictionary Based Edge Blocking off for email@example.com?
Back to case 7. So why not simply turn off the Directory Based Edge Blocking for firstname.lastname@example.org by configuring that domain as Internal Relay and not Authoritative. We can do that by going to Exchange Online> Accepted Domains, and then we can make the @contoso.onmicrosoft.com as Internal Relay domain and not Authoritative. This will turn off the Dictionary Based Edge Blocking. Going back to case 7 above, when EOP receives a message going to StrangeGuid@contoso.onmicrosoft.com, it will not bother asking Azure AD anything, and will just deliver the message to Exchange Online. Exchange Online knows this address, as it is one of many arbitration mailboxes in Office 365, and everything will work just fine.
Wake Up !! This should never ever been done. The domain @contoso.onmicrosoft.com should always be authoritative domain or other stuff will stop working.
The first thing that will not work the moment you switch that domain to Internal Relay is the Hosted Voice Mail. Hosted Voice Mail REQUIRES that the domain @contoso.onmicrosoft.com to be Authoritative. There is no clear reason why this is required, but it took us long troubleshooting time to figure this out and fix the hosted voice mail system.
Note: Hosted Voice Mail means that we have people with Office 365 mailboxes and using the on-premises Skype for Business telephony, and we need the Skype for Business on-premises to deliver voice mail for Office 365 through Skype federation with Exchange Online. This requires the domain @contoso.onmirosoft.com to be configured as Authoritative domain in Exchange Online Admin Console.
Other Solution, creating contacts
So the other option that some Microsoft support engineer proposed is to create a contact in our Exchange Online system, mail enabled contact with an email address StrangeGuid@contoso.onmicrosoft.com.
StrangeGuid@contoso.onmicrosoft.com is the smtp address of the Exchange Online arbitration mailbox.
Now going back to case 7 above, when Bob sends email to StrangeGuid@contoso.onmicrosoft.com and since that domain is authoritative and the Dictionary Based Edge Blocking logic will kick, EOP will try to match that recipient address with any recipient defined in Azure AD, and it will found that mail enabled contact that we just made, and it will deliver the mail to Exchange Online that knows how to deal with it.
This will work will except that there are many arbitration mailboxes in Exchange Online and we do not know how to query them. The best guess is to try to simulate the Case 7 over and over and for each new arbitration mailbox Online that we detect, we shall create a mail enabled contact in Exchange Online to trick EOP. This does not work well in the long run.
The best thing to do is to try to scope the problem. We know we have seven out of the eight cases working just fine, and we have only one case not working fine. Only when the moderator is on-premises we have an issue.
We also do not have any issues with on-premises dynamic groups, only non-dynamic groups with on-premises moderator is where we need to look at.
One solution could be migrate the mailbox of all on-premises moderators for non dynamic groups. This can play well, but needs some time which we cannot have now.
The other solution is this. Remember why we do not have problem with on-premises dynamic groups even though they have on-premises moderators? Well, because Exchange Online do not know about them, because they are not synced to Azure AD in the first place, which in place triggers the email to go to the on-premises arbitration mailbox, which works just fine.
The proposed solution is to query our directory for all non-dynamic on-premises moderated groups, that have on-premises moderators, and put them in an OU that is not synced to Azure AD, and then create Exchange contacts for them in Exchange Online. This will solve the whole issue for now until all moderators are in Office 365.
The only bad thing about this is that Office 365 users will not be able to expand those groups because they appear as contacts, and will not see the mail tip that the group is moderated (which we can mimic by using mail tips). This will work well I believe for the time being.
Now, when Office 365 users trying to send to one of those groups, they will open address book, and see a group called email@example.com. The fact is this is a mail enabled contact, so Exchange Online will not know it is moderated, and will just send it to Exchange on-premises, which will send it to the on-premises arbitration mailbox and everything will work just fine.
Hybrid email moderation in Office 365 is hard to troubleshoot and figure out unless you dig deep to how things work. I found my self going crazy trying to figure out this hybrid email moderation in Office 365 issues, and I felt I have to share with. Please share the post if you find it useful. Please find my Exchange group moderation report to query all your moderated groups.