Email analysis in investigations for Microsoft Defender for Office 365

During the automated investigation of alerts, Microsoft Defender for Office 365 analyzes the original email for threats and identifies other emails that are related to the original email and potentially part of an attack. This analysis is important because email attacks rarely consist of a single email.

The automated investigation’s email analysis identifies email clusters using attributes from the original email to query for emails sent and received by your organization. This is similar to a security operations analyst would hunt for the related emails in Explorer or Advanced Hunting. Several queries are used to identify matching emails because attackers typically morph the email parameters to avoid security detection. The clustering analysis performs these checks to determine how to handle emails involved in the investigation:

• The email analysis creates queries (clusters) of emails using attributes from the original email – sender values (IP address, sending domain) and contents (subject, cluster ID) in order to find emails that are related.
• If analysis of the original email’s URLs and files identifies that some are malicious (that is, malware or phish), then it will also create queries or clusters of emails containing the malicious URL or file.
• Email clustering analysis counts the threats associated with the matching emails in the cluster to determine whether the emails are malicious, suspicious, or have no clear threats. If the cluster of emails matching the query has a sufficient amount of spam, normal phish, high confidence phish or malware threats, the email cluster gets that threat type applied to it.
• The email clustering analysis also checks the latest delivery location of the original email and emails in the email clusters to help identify if the emails potentially still need removal or have already been remediated or prevented. This analysis is important because attackers morph malicious content plus security policies and protection may vary between mailboxes. This capability leads to situations where malicious content may still sit in mailboxes, even though one or more malicious emails have been detected and removed by zero-hour auto purge (ZAP).
• Email clusters that are considered malicious due to malware, high confidence phish, malicious files, or malicious URLs threats will get a pending action to soft delete the emails when there are still in the cloud mailbox (inbox or junk folder). If malicious emails or email clusters are only “Not In Mailbox” (blocked, quarantined, failed, soft deleted, etc.) or “On-premises/External” with none in the cloud mailbox, then no pending action will be set up to remove them.
• If any of the email clusters are determined to be malicious, then the threat identified by the cluster will get applied back to the original email involved in the investigation. This behavior is similar to a security operations analyst using email hunting results to determine the verdict of an original email based on matching emails. This result ensures that regardless of whether an original email’s URLs, files, or source email indicators are detected or not, the system can identify malicious emails that are potentially evading detection through personalization, morphing, evasion, or other attacker techniques.
• In the user compromise investigation, additional email clusters are created to identify potential email issues created by the mailbox. This process includes a clean email cluster (good emails from user, potential data exfiltration, and potential command/control emails), suspicious email clusters (emails containing spam or normal phish) and malicious email clusters (emails containing malware or high confidence phish). These email clusters provide security operations analysts data to determine what other problems may need to be addressed from a compromise, and visibility on which emails may have triggered the original alerts (for example, phish/spam that triggered user sending restrictions)

Email clustering analysis via similarity and malicious entity queries ensures that email problems are fully identified and cleaned up, even if only one email from an attack gets identified. You can use links from the email cluster details side panel views to open the queries in Explorer or Advanced Hunting to perform deeper analysis and change the queries if needed. This capability enables manual refinement and remediation if you find the email cluster’s queries too narrow or too broad (including unrelated emails).

How to measure email marketing success

Step 1. Understand everything you can measure in email marketing

Step 2. Set up goals

Different campaigns will have different goals. You should not expect every email to bring you revenue. Sometimes, all you are looking for is to re-engage with the customer and explain your unique selling propositions, hopefully leading to purchase later.

Step 3. Define KPIs

Once you know the goals for your campaign, you can start with defining KPIs. It’s important to focus on both “positive” KPIs – such as revenue uplift or open rate uplift, while also looking at minimizing “negative” KPIs – such as keeping your unsubscribe rate below a certain level.

Step 4. Track metrics

EXPONEA NOTE:

Step 5. Evaluate

Evaluating email metrics is quite tricky – some of them need to be looked at daily, some weekly, and others you can only check monthly.

Check these daily : open rate, open rate variation, bounce rate, soft bounce rate.

Step 6. Improve

Knowledge of benchmarks will help you prioritize what needs to be improved. If you don’t like your conversion rates for a newsletter, but you’re still beating benchmarks, then there might be another area to focus on where you can get a bigger improvement for the same effort.

It’s completely normal to feel overwhelmed by the sheer size of the reporting – there are lots of different metrics, and it’s difficult to even know where to start. If you feel this way, then just start at the beginning of the funnel: your customers, your audience criteria, and your email frequency. Once you’re happy with this part of the funnel and you’re getting results you like, you can move on to the next steps: delivery rate, open rate, click rate, and eventually revenue.

Remember that you’ll need to be patient. Big changes, especially in email, take time. Be patient and set reasonable timelines, and don’t expect miracles after two campaign sends. Keep this in mind when communicating with your team or senior stakeholders. Email marketing is a complex system, and it depends on more than just your performance. There are external factors at play. You need to test and work towards finding what works for you, your goals, your customers, the mailbox providers, and those trying to fight spam

Email Marketing Metrics

Email audience trend

As you gain subscribers, you also lose subscribers – this is a natural process. Knowing that your audience is declining, and knowing this early, will help you prevent bigger revenue issues 3 months down the line – it enables you to take actions, to understand why subscribers are leaving and what you need to change.

Active audience trend

It’s also vital to monitor the growth or decline of your active audience as well. These are the subscribers that engage with most of your emails, bring the most revenue, and according to best practices , are the only ones with whom you should communicate with a high frequency.

Delivery rate

Delivery of an email is usually the first thing measured after an email is sent – it means that your email was accepted by the recipient server. In the case of a bigger deliverability issue, you would see your delivery rate drop – this might indicate that you are being blocked completely.

As long as your database is clean, you should be achieving a 99% delivery rate or higher, although you might see lower delivery rates on programs such as double opt in , or a welcome program, where new email addresses enter your list.

Hard bounce rate

The bounce rate informs you about hard bounced emails: emails that are permanently unreachable. You should automatically exclude any bounced addresses from your list. Failure to do so will lead to your emails being blocked completely.

A high bounce rate can be caused either by already being blacklisted, or by sending to inactive emails (this is more common). You can find out what’s happening by looking at status messages or code that was sent back to you.

Soft bounce rate

Soft bounce rate informs you about temporary issues in your emailing. This doesn’t mean it’s not important to monitor, as it can hint at bigger issues. If you ignore these warnings, you will eventually encounter hard bounces and a drop in delivery rates.

Frequent causes for soft bounces are temporary blacklisting, full mailboxes, connection errors, and greylisting. To avoid these, make sure your email list is clean and that you use best sending practices . You should aim for a less than 0.5% soft bounce rate.

If you encounter a big spike in soft bounces, it could be a mistake or bad filtering on the recipient domain side. You can try solving this with your ESP, who will be able to reach out to the recipient domain.

Open rate

Open rate is the 2nd most important deliverability metric because it shows you both that emails are reaching the customers inbox, and that either the brand or the subject line caught their attention. Most brands calculate this as unique opens, since counting all opens can skew the results.

The open rate is determined by displaying a 1px x 1px image at the top of the email. When the email is opened, the image is downloaded, which lets the tool know who opened an email and when it happened. However, there is a downside to this method: if images are blocked by the email client, there is no way to track opens.

Total open rate

This metric, when compared to the previous open rate metric, will help you understand how many customers are coming back to reopen an email. This is helpful for finding out whether customers are coming back to emails with, for example, USPs or loyalty program information. If they are, you might think about sharing that information in an easier to find way, so they’re not searching for old emails.

Open rate variation

Resources:

https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/email-analysis-investigations
https://exponea.com/blog/email-marketing-analytics-metrics-goals-kpis/
https://www.getresponse.com/blog/email-marketing-metrics
https://docs.microsoft.com/en-us/microsoft-365/admin/usage-analytics/usage-analytics